How Can We Know What Language Models Know?
Open accessThe DOI matches its Crossref record; 51 of the 52 references it cites were found in a registry or an index.
Abstract
Recent work has presented intriguing results examining the knowledge contained in language models (LMs) by having the LM fill in the blanks of prompts such as “ Obama is a __ by profession”. These prompts are usually manually created, and quite possibly sub-optimal; another prompt such as “ Obama worked as a __ ” may result in more accurately predicting the correct profession. Because of this, given an inappropriate prompt, we might fail to retrieve facts that the LM does know, and thus any given prompt only provides a lower bound estimate of the knowledge contained in an LM. In this paper, we attempt to more accurately estimate the knowledge contained in LMs by automatically discovering better prompts to use in this querying process. Specifically, we propose mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different prompts. Extensive experiments on the LAMA benchmark for extracting relational knowledge from LMs demonstrate that our methods can improve accuracy from 31.1% to 39.6%, providing a tighter lower bound on what LMs know. We have released the code and the resulting LM Prompt And Query Archive (LPAQA) at https://github.com/jzbjyb/LPAQA .
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The ability of Large Language Models to retrieve factual knowledge as explicit knowledge bases [53-56, 56-61] has been stirred discussed, which presents an opportunity to construct more comprehensive knowledge graphs within recommender systems.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [56]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“even experienced prompt designers may fail to manually discover optimal prompts [52].”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [52]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Beyond this, we found models trained with Toolformer to often be sensitive to the exact wording of their input when deciding whether or not to call an API; this is perhaps unsurprising given that LMs are known to be very sensitive to the prompt they are provided with in both zeroand few-shot settings (Jiang et al., 2020;Schick and Schütze, 2021a).”
From Toolformer: Language Models Can Teach Themselves to Use Tools (2023) · cites this paper as Jiang et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“There is some evidence to suggest that formats that occur commonly in the training data will lead to better performance (Jiang et al., 2020).”
From The Prompt Report: A Systematic Survey of Prompt Engineering Techniques (2024) · cites this paper as Jiang et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Jiang et al. [52]'s Mine approach is a mining-based method to automatically find templates given a set of training inputs x and outputs y.”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [52]
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“Prompts where the hidden vectors learned through a language modeling objective are then used in downstream language understanding systems (Dai and Le, 2015; Melamud et al., 2016; Peters et al., 2018; Devlin et al., 2019).”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
A Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1014 · Cited by 1,926
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834
ERNIE: Enhanced Language Representation with Informative Entities
Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., & Liu, Q. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1139 · Cited by 1,482 · cited 5×
“Similar extensions have been applied to pre-trained LMs like BERT, where contextualized representations are enhancedwith entity embeddings (Zhang et al., 2019; Peters et al., 2019; Pörner et al., 2019).”
MethodInfluentialOpen information extraction from the web
Banko, M., Cafarella, M., Soderland, S. G., Broadhead, M., & Etzioni, O. (2007) · International Joint Conference on Artificial Intelligence · Cited by 1,398
BERT Rediscovers the Classical NLP Pipeline
Tenney, I., Das, D., & Pavlick, E. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1452 · Cited by 1,307
(2019) · doi:10.18653/v1/p19-1356 · Cited by 1,291 · cited 1×
“…a broad suite of NLP tasks are used to analyze both syntactic and semantic properties, providing evidence that contextualized representations encode linguistic knowledge in different layers (Hewitt and Manning, 2019; Tenney et al., 2019a; Tenney et al., 2019b; Jawahar et al., 2019; Goldberg, 2019).”
BackgroundSnowball: extracting relations from large plain-text collections
Agichtein, E., & Gravano, L. (2000) · Proceedings of the fifth ACM conference on Digital libraries · doi:10.1145/336597.336644 · Cited by 1,147
Research works printed but not linked: 7
- Andrew M. Dai and Quoc V. Le. 2015. Semisupervised sequence learning. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada , pages 3079-3087.
- Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A diversitypromoting objective function for neural conversation models. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016 , pages 110-119.
- Gábor Melis, Chris Dyer, and Phil Blunsom. 2018. On the state of the art of evaluation in neural language models. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings .
- Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018. Regularizing and optimizing LSTM language models. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings .
- 1-6, 2018, Volume 1 (Long Papers) , pages 22272237.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog , 1(8).
- Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019b. What do you learn from context? Probing for sentence structure in contextualized word representations. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 .
Cited by
Items 401–500 of 1,040
PTCAS: Prompt tuning with continuous answer search for relation extraction
Chen, Y., Shi, B., & Xu, K. (2024) · Information Sciences · doi:10.1016/j.ins.2023.120060 · Cited by 16
Towards fair decision: A novel representation method for debiasing pre-trained models
He, J., Lin, N., Bai, Q., Liang, H., Zhou, D. M., & Yang, A. (2024) · Decision Support Systems · doi:10.1016/j.dss.2024.114208 · Cited by 19
Semantic-Oriented Visual Prompt Learning for Diabetic Retinopathy Grading on Fundus Images
Zhang, Y., Ma, X., Huang, K., Li, M., & Heng, P. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3383827 · Cited by 21
ChatGPT como herramienta para mejorar la expresión escrita en inglés como lengua extranjera
de Valencia, U., & Vayá, A. R. (2024) · Íkala Revista de Lenguaje y Cultura · doi:10.17533/udea.ikala.354584 · Cited by 23
Li, Y., Du, M., Song, R., Wang, X., Sun, M., & Wang, Y. (2024) · Artificial Intelligence · doi:10.1016/j.artint.2024.104143 · Cited by 31
Zero-Shot Temporal Action Detection by Learning Multimodal Prompts and Text-Enhanced Actionness
Raza, A., Yang, B., & Zou, Y. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3414275 · Cited by 24
Distilroberta2gnn: a new hybrid deep learning approach for aspect-based sentiment analysis
Alhadlaq, A., & Altheneyan, A. S. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.2267 · Cited by 23
AuditLLM: A Tool for Auditing Large Language Models Using Multiprobe Approach
Amirizaniani, M., Martin, E., Roosta, T. G., Chadha, A. R., & Shah, C. S. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679222 · Cited by 33
Do Language Models Know the Way to Rome?
Liétard, B., Abdou, M., & Søgaard, A. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.40 · Cited by 32
A Composable Generative Framework Based on Prompt Learning for Various Information Extraction Tasks
Kan, Z., Feng, L., Yin, Z., Qiao, L., Qiu, X., & Li, D. (2023) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2023.3278977 · Cited by 19
Analyzing the Forgetting Problem in the Pretrain-Finetuning of Dialogue Response ModelsPreprint
He, T., Liu, J., Cho, K., Ott, M., Liu, B., Glass, J. R., & Peng, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.07117 · Cited by 21
Automating the Enterprise with Foundation Models
Wornow, M., Narayan, A., Opsahl-Ong, K., McIntyre, Q., Shah, N., & Ré, C. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3681954.3681964 · Cited by 15
How Reliable are Model Diagnostics?
Aribandi, V., Tay, Y., & Metzler, D. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.155 · Cited by 15
Development of Linguistic Competence in English for Specific Purposes Through ChatGPT: A Case Study
Enamorado, M. Á. J. (2025) · Journal of language and Education · doi:10.17323/jle.2025.23745 · Cited by 21
How Reliable Are AI-Generated-Text Detectors? An Assessment Framework Using Evasive Soft Prompts
Kumarage, T., Sheth, P., Moraffah, R., Garland, J., & Liu, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.94 · Cited by 43
Defense-Prefix for Preventing Typographic Attacks on CLIP
Azuma, H., & Matsui, Y. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00392 · Cited by 52
Zero-shot Triplet Extraction by Template Infilling
Kim, B., Iso, H., Bhutani, N., Junior, E. R. H., Nakashole, N., & Mitchell, T. M. (2023) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2023.ijcnlp-main.18 · Cited by 30
TaskLAMA: Probing the Complex Task Understanding of Language Models
Yuan, Q., Kazemi, M., Xu, X., Noble, I., Imbrasaitė, V., & Ramachandran, D. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29918 · Cited by 32
Graph-Enhanced Prompt Learning for Personalized Review Generation
Qu, X., Wang, Y., Zhao, L., & Gao, J. (2024) · Data Science and Engineering · doi:10.1007/s41019-024-00252-z · Cited by 18
CLOVA: A Closed-LOop Visual Assistant with Tool Usage and Update
Gao, Z. F., Du, Y., Zhang, X., Ma, X., Han, W., Zhu, S. C., & Li, Q. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01259 · Cited by 72
MV-CLIP: Multi-View CLIP for Zero-Shot 3D Shape Recognition
Song, D., Fu, X., Liu, N., Nie, W., Li, W., Wang, L., Yang, Y., & Liu, A.-A. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3551084 · Cited by 29
TaxonPrompt: Taxonomy-aware curriculum prompt learning for few-shot event classification
Song, C., Cai, F., Wang, M., Zheng, J., & Shao, T. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110290 · Cited by 18
Automating the Enterprise with Foundation Models
Wornow, M., Narayan, A., Opsahl-Ong, K., McIntyre, Q., Shah, N. H., & Re, C. (2024) · arXiv · doi:10.48550/arxiv.2405.03710 · Cited by 15
Knowledge of Knowledge: Exploring Known-Unknowns Uncertainty with Large Language Models
Amayuelas, A., Wong, K., Pan, L., Chen, W., & Wang, W. (2023) · arXiv · doi:10.48550/arxiv.2305.13712 · Cited by 15
Apidianaki, M., & Soler, A. G. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.7 · Cited by 18
Analyzing BERT’s Knowledge of Hypernymy via Prompting
Hanna, M., & Mareček, D. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.20 · Cited by 40
Enhancing Entity Representations with Prompt Learning for Biomedical Entity Linking
Zhu, T., Qin, Y., Chen, Q., Hu, B., & Xiang, Y. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/560 · Cited by 15
Exploring better image captioning with grid features
Yan, J., Xie, Y., Guo, Y., Wei, Y., & Luan, X. (2024) · Complex & Intelligent Systems · doi:10.1007/s40747-023-01341-8 · Cited by 17
KeyEE: Enhancing Low-Resource Generative Event Extraction with Auxiliary Keyword Sub-Prompt
Duan, J., Liao, X., An, Y., & Wang, J. (2024) · Big Data Mining and Analytics · doi:10.26599/bdma.2023.9020036 · Cited by 15
Li, J., Cotterell, R., & Sachan, M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.84 · Cited by 27
On Measuring Social Biases in Prompt-Based Multi-Task Learning
Akyürek, A. F., Paik, S., Kocyigit, M. Y., Akbıyık, S., Runyun, Ş. L., & Wijaya, D. T. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.42 · Cited by 33
PromptGen: Automatically Generate Prompts using Generative Models
Zhang, Y., Fei, H., Li, D., & Li, P. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.3 · Cited by 35
Prediction of Vulnerability Characteristics Based on Vulnerability Description and Prompt Learning
Li, X., Ren, X., Xue, Y., Xing, Z., & Sun, J. (2023) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner56733.2023.00062 · Cited by 20
Pre-training Language Models with Deterministic Factual Knowledge
Li, S., Li, X., Shang, L., Sun, C., Liu, B., Ji, Z., Jiang, X., & Liu, Q. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.764 · Cited by 20
Jiang, H., Shi, S., Yao, Y., Jiang, C., & Li, Q. (2025) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2025.3633887 · Cited by 18
Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models
Youssef, P., Koraş, O. A., Li, M., Schlötterer, J., & Seifert, C. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1043 · Cited by 52
Muthusamy, V., Rizk, Y., Kate, K., Venkateswaran, P., Isahagian, V., Gulati, A., & Dube, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.461 · Cited by 40
Cross-Domain Fake News Detection Using a Prompt-Based Approach
Alghamdi, J., Lin, Y., & Luo, S. (2024) · Future Internet · doi:10.3390/fi16080286 · Cited by 16
GADformer: A Transparent Transformer Model for Group Anomaly Detection on Trajectories
Lohrer, A., Malik, D., Zelenka, C., & Kröger, P. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650999 · Cited by 13
PromptMap: An Alternative Interaction Style for AI-Based Image Generation
Adamkiewicz, K., Woźniak, P. W., Dominiak, J., Romanowski, A., Karolus, J., & Frolov, S. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712150 · Cited by 25
Prompt Engineering for Large Language Model-Assisted Inductive Thematic Analysis
Khalid, M. T., & Witmer, A. (2025) · Social Science Computer Review · doi:10.1177/08944393251388098 · Cited by 24
Static Embeddings as Efficient Knowledge Bases?
Dufter, P., Kassner, N., & Schütze, H. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.186 · Cited by 25
Answer-level Calibration for Free-form Multiple Choice Question Answering
Kumar, S. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.49 · Cited by 29
Contextual Information and Commonsense Based Prompt for Emotion Recognition in Conversation
Yi, J., Yang, D., Yuan, S., Cao, K., Zhang, Z., & Xiao, Y. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-26390-3_41 · Cited by 18
CP-Rec: Contextual Prompting for Conversational Recommender Systems
Chen, K., & Sun, S. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26487 · Cited by 22
Don’t Prompt, Search! Mining-based Zero-Shot Learning with Language Models
van de Kar, M., Xia, M., Chen, D., & Artetxe, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.509 · Cited by 21
The ACL OCL Corpus: Advancing Open Science in Computational Linguistics
Rohatgi, S., Qin, Y., Aw, B., Unnithan, N., & Kan, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.640 · Cited by 44
Liu, Y., An, C., & Qiu, X. (2023) · Frontiers of Computer Science · doi:10.1007/s11704-023-3131-8 · Cited by 32
Chen, Y., Cao, P., Chen, Y., Liu, K., & Zhao, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29735 · Cited by 12
Large language models for PHM: a review of optimization techniques and applications
Yu, T., Tang, J., Yu, Q., Li, L., Liu, Y., & Poler, R. (2025) · Autonomous Intelligent Systems · doi:10.1007/s43684-025-00100-5 · Cited by 15
Probing Simile Knowledge from Pre-trained Language Models
Chen, W., Chang, Y., Zhang, R., Pu, J., Chen, G., Zhang, L., Xi, Y., Chen, Y., & Su, C. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.404 · Cited by 21
Are the BERT family zero-shot learners? A study on their potential and limitations
Wang, Y., Wu, L., Li, J., Liang, X., & Zhang, M. (2023) · Artificial Intelligence · doi:10.1016/j.artint.2023.103953 · Cited by 20
Cross-domain coreference modeling in dialogue state tracking with prompt learning
Xu, H.-D., Mao, X.-L., Yang, P., Sun, F., & Huang, H. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111189 · Cited by 13
From Raw Video to Pedagogical Insights: A Unified Framework for Student Behavior Analysis
Yu, Z., Xie, M., Gao, J., Liu, T., & Fu, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i21.30371 · Cited by 16
Knowledge-Enriched Prompt for Low-Resource Named Entity Recognition
Hou, W., Zhao, W., Liu, X., & Guo, W. (2024) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3659948 · Cited by 15
SKR-QA: Semantic ranking and knowledge revise for multi-choice question answering
Ren, M., Huang, H., & Gao, Y. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.06.076 · Cited by 12
Decomposed Two-Stage Prompt Learning for Few-Shot Named Entity Recognition
Ye, F., Huang, L. P., Liang, S., & Chi, K. (2023) · Information · doi:10.3390/info14050262 · Cited by 23
Ma, Z., Yu, Z., Li, J., & Li, G. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26569 · Cited by 18
Prompt Conditioned VAE: Enhancing Generative Replay for Lifelong Learning in Task-Oriented Dialogue
Zhao, Y., Zheng, Y., Tian, Z., Gao, C., Sun, J., & Zhang, N. L. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.766 · Cited by 27
Enhancing Zero-Shot Stance Detection with Contrastive and Prompt Learning
Yao, Z., Yang, W., & Wei, F. (2024) · Entropy · doi:10.3390/e26040325 · Cited by 16
Understanding the Source of Semantic Regularities in Word Embeddings
Chiang, H.-Y., Camacho-Collados, J., & Pardos, Z. A. (2020) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2020.conll-1.9 · Cited by 21
On Transferability of Prompt Tuning for Natural Language UnderstandingPreprint
Su, Y., Wang, X., Qin, Y., Chan, C.-M., Lin, Y., Liu, Z., Li, P., Li, J., Hou, L., Sun, M., et al. (2021) · arXiv (Cornell University) · Cited by 17
How Reliable Are AI-Generated-Text Detectors? An Assessment Framework Using Evasive Soft Prompts
Kumarage, T., Sheth, P., Moraffah, R., Garland, J., & Liu, H. (2023) · arXiv · doi:10.48550/arxiv.2310.05095 · Cited by 11
“Prompt tuning is a widely used approach for guiding PLMs to generate desired outputs (Jiang et al., 2020).”
MethodDSP: Discriminative Soft Prompts for Zero-Shot Entity and Relation Extraction
Lv, B., Liu, X., Dai, S., Liu, N., Yang, F., Luo, P., & Yu, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.339 · Cited by 23
Training Models to Generate, Recognize, and Reframe Unhelpful Thoughts
Maddela, M., Ung, M., Xu, J., Madotto, A., Foran, H., & Boureau, Y.-L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.763 · Cited by 60
Zero-shot Approach to Overcome Perturbation Sensitivity of Prompts
Chakraborty, M., Kulkarni, A., & Li, Q. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.313 · Cited by 23
Qin, Y., & Irshad, A. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.1895 · Cited by 11
A Teleological Approach to Information Systems Design
Fumagalli, M., Ferrario, R., & Guizzardi, G. (2024) · Minds and Machines · doi:10.1007/s11023-024-09673-1 · Cited by 12
Jalali, M., Luo, Y., Caulfield, L., Sauter, E., Nefedov, A., & Wöll, C. (2024) · Materials Today Communications · doi:10.1016/j.mtcomm.2024.109801 · Cited by 22
Generalization Boosted Adapter for Open-Vocabulary Segmentation
Xu, W., Wang, C., Feng, X., Xu, R., Huang, L., Zhang, Z., Guo, L., & Xu, S. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3454227 · Cited by 21
The Factuality of Large Language Models in the Legal Domain
Hamdani, R. E., Bonald, T., Malliaros, F. D., Holzenberger, N., & Suchanek, F. M. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679961 · Cited by 25
Empowering IoT-Based Autonomous Driving via Federated Instruction Tuning With Feature Diversity
Chen, J., He, J., Chen, F., Lv, Z., Tang, J., & Jia, Y. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3518615 · Cited by 19
How Relevant Are Selectional Preferences for Transformer-based Language Models?
Metheniti, E., de Cruys, T. V., & Hathout, N. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.109 · Cited by 20
ContrastNER: Contrastive-based Prompt Tuning for Few-shot NER
Layegh, A., Payberah, A. H., Soylu, A., Roman, D., & Matskin, M. (2023) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac57700.2023.00038 · Cited by 21
SwitchPrompt: Learning Domain-Specific Gated Soft Prompts for Classification in Low-Resource Domains
Goswami, K., Lange, L., Araki, J., & Adel, H. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.197 · Cited by 23
Self-citationDialogue for Prompting: A Policy-Gradient-Based Discrete Prompt Generation for Few-Shot Learning
Li, C., Liu, X., Wang, Y., Li, D., Lan, Y., & Shen, C. Q. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29809 · Cited by 26
Adaptive Prompt Routing for Arbitrary Text Style Transfer with Pre-trained Language Models
Liu, Q., Qin, J., Ye, W., Mou, H. L., He, Y., & Wang, K. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29832 · Cited by 33
Assessing the Effectiveness of Vulnerability Detection via Prompt Tuning: An Empirical Study
Lu, G., Ju, X., Chen, X., Yang, S., Chen, L., & Shen, H. (2023) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec60848.2023.00052 · Cited by 19
Dissecting Generation Modes for Abstractive Summarization Models via Ablation and Attribution
Xu, J., & Durrett, G. (2021) · doi:10.18653/v1/2021.acl-long.539 · Cited by 19
How Many Data Samples is an Additional Instruction Worth?
Puri, R. S., Mishra, S. R., Parmar, M., & Baral, C. R. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.77 · Cited by 22
RELiC: Retrieving Evidence for Literary Claims
Thai, K., Chang, Y., Krishna, K., & Iyyer, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.517 · Cited by 28
Improving Commonsense in Vision-Language Models via Knowledge Graph Riddles
Ye, S., Xie, Y., Chen, D., Yi‐chong, X., Yuan, L., Zhu, C., & Liao, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00259 · Cited by 29
Neural Knowledge Bank for Pretrained Transformers
Dai, D., Jiang, W., Dong, Q., Lyu, Y., & Sui, Z. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44696-2_60 · Cited by 26
Thangka Image Captioning Based on Semantic Concept Prompt and Multimodal Feature Optimization
Hu, W., Qiao, L., Kang, W., & Shi, X. (2023) · Journal of Imaging · doi:10.3390/jimaging9080162 · Cited by 13
Hybrid API Migration: A Marriage of Small API Mapping Models and Large Language Models
Zhou, B., Wang, X., Xu, S., Yao, Y., Pan, M., Xu, F., & Ma, X. (2023) · Asia-Pacific Symposium on Internetware · doi:10.1145/3609437.3609466 · Cited by 18
Self-citationClickbait Detection via Prompt-Tuning With Titles Only
Wang, Y., Zhu, Y. Q., Li, Y., Qiang, J., Yuan, Y., & Wu, X. (2024) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2024.3406418 · Cited by 11
Knowledge-injected prompt learning for actionable information extraction from crisis-related tweets
Gao, W., Zheng, C., Zhu, X., Deng, H., Wang, Y., & Hu, G. (2024) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2024.109398 · Cited by 12
Domain adaptation for textual adversarial defense via prompt-tuning
Li, Z., Zhu, Y. Q., Hua, C., Li, Y., Yuan, Y.-H., & Qiang, J. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129192 · Cited by 11
A Slim Prompt-Averaged Consistency prompt learning for vision–language model
He, S., Wang, S., & Long, S. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113011 · Cited by 10
A multi-task learning framework with prompt-based hybrid alignment for vision-language model
Han, J., Wei, Z., Ding, C., & Wang, C. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130121 · Cited by 10
Exploring Strategies for Generalizable Commonsense Reasoning with Pre-trained Models
Ma, K., Ilievski, F., Francis, J., Ozaki, S., Nyberg, E., & Oltramari, A. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.445 · Cited by 23
Enhancing Class Understanding Via Prompt-Tuning For Zero-Shot Text Classification
Dan, Y., Zhou, J., Chen, Q., Bai, Q., & He, L. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746200 · Cited by 15
Position-based Prompting for Health Outcome Generation
Abaho, M., Bollegala, D., Williamson, P. R., & Dodd, S. R. (2022) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2022.bionlp-1.3 · Cited by 15
PromptORE - A Novel Approach Towards Fully Unsupervised Relation Extraction
Genest, P.-Y., Portier, P.-É., Egyed‐Zsigmond, E., & Goix, L.-W. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557422 · Cited by 25
Causality-aware Concept Extraction based on Knowledge-guided Prompting
Yuan, S., Yang, D., Liu, J., Tian, S., Liang, J., Xiao, Y., & Xie, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.514 · Cited by 20
Emergent Modularity in Pre-trained Transformers
Zhang, Z., Zeng, Z., Lin, Y., Xiao, C., Wang, X., Han, X., Liu, Z., Xie, R., Sun, M., & Zhou, J. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.250 · Cited by 40
Can ChatGPT Understand Causal Language in Science Claims?
Kim, Y., Guo, L., Yu, B., & Li, Y. (2023) · Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA) · doi:10.18653/v1/2023.wassa-1.33 · Cited by 24
Prompted and integrated textual information enhancing aspect-based sentiment analysis
Shi, X., Hu, M. Q., Ren, F., Shi, P., Deng, J., & Tang, Y. (2023) · Journal of Intelligent Information Systems · doi:10.1007/s10844-023-00805-0 · Cited by 11
Xu, B., Zheng, S., & Jin, Q. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612484 · Cited by 20
Revisiting the Knowledge Injection Frameworks
Fu, P., Zhang, Y., Wang, H., Qiu, W., & Zhao, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.677 · Cited by 26
Metadata
- Authors
- Jiang, Z., Xu, F. F., Araki, J., & Neubig, G.
- Affiliations
- Carnegie Mellon University (US)
- Robert Bosch (United States) (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Pages
- 423-438
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2991382858
- Cited by
- 1,002
- References
- 52
- Citations by year
- 2026: 76 · 2025: 169 · 2024: 224 · 2023: 285 · 2022: 144 · 2021: 78 · 2020: 25
- FWCI
- 52.35
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1815287)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- prompt generation, ensemble methods
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.