How Can We Know What Language Models Know?
Open accessThe DOI matches its Crossref record; 51 of the 52 references it cites were found in a registry or an index.
Abstract
Recent work has presented intriguing results examining the knowledge contained in language models (LMs) by having the LM fill in the blanks of prompts such as “ Obama is a __ by profession”. These prompts are usually manually created, and quite possibly sub-optimal; another prompt such as “ Obama worked as a __ ” may result in more accurately predicting the correct profession. Because of this, given an inappropriate prompt, we might fail to retrieve facts that the LM does know, and thus any given prompt only provides a lower bound estimate of the knowledge contained in an LM. In this paper, we attempt to more accurately estimate the knowledge contained in LMs by automatically discovering better prompts to use in this querying process. Specifically, we propose mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different prompts. Extensive experiments on the LAMA benchmark for extracting relational knowledge from LMs demonstrate that our methods can improve accuracy from 31.1% to 39.6%, providing a tighter lower bound on what LMs know. We have released the code and the resulting LM Prompt And Query Archive (LPAQA) at https://github.com/jzbjyb/LPAQA .
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The ability of Large Language Models to retrieve factual knowledge as explicit knowledge bases [53-56, 56-61] has been stirred discussed, which presents an opportunity to construct more comprehensive knowledge graphs within recommender systems.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [56]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“even experienced prompt designers may fail to manually discover optimal prompts [52].”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [52]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Beyond this, we found models trained with Toolformer to often be sensitive to the exact wording of their input when deciding whether or not to call an API; this is perhaps unsurprising given that LMs are known to be very sensitive to the prompt they are provided with in both zeroand few-shot settings (Jiang et al., 2020;Schick and Schütze, 2021a).”
From Toolformer: Language Models Can Teach Themselves to Use Tools (2023) · cites this paper as Jiang et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“There is some evidence to suggest that formats that occur commonly in the training data will lead to better performance (Jiang et al., 2020).”
From The Prompt Report: A Systematic Survey of Prompt Engineering Techniques (2024) · cites this paper as Jiang et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Jiang et al. [52]'s Mine approach is a mining-based method to automatically find templates given a set of training inputs x and outputs y.”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [52]
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“Prompts where the hidden vectors learned through a language modeling objective are then used in downstream language understanding systems (Dai and Le, 2015; Melamud et al., 2016; Peters et al., 2018; Devlin et al., 2019).”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
A Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1014 · Cited by 1,926
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834
ERNIE: Enhanced Language Representation with Informative Entities
Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., & Liu, Q. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1139 · Cited by 1,482 · cited 5×
“Similar extensions have been applied to pre-trained LMs like BERT, where contextualized representations are enhancedwith entity embeddings (Zhang et al., 2019; Peters et al., 2019; Pörner et al., 2019).”
MethodInfluentialOpen information extraction from the web
Banko, M., Cafarella, M., Soderland, S. G., Broadhead, M., & Etzioni, O. (2007) · International Joint Conference on Artificial Intelligence · Cited by 1,398
BERT Rediscovers the Classical NLP Pipeline
Tenney, I., Das, D., & Pavlick, E. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1452 · Cited by 1,307
(2019) · doi:10.18653/v1/p19-1356 · Cited by 1,291 · cited 1×
“…a broad suite of NLP tasks are used to analyze both syntactic and semantic properties, providing evidence that contextualized representations encode linguistic knowledge in different layers (Hewitt and Manning, 2019; Tenney et al., 2019a; Tenney et al., 2019b; Jawahar et al., 2019; Goldberg, 2019).”
BackgroundSnowball: extracting relations from large plain-text collections
Agichtein, E., & Gravano, L. (2000) · Proceedings of the fifth ACM conference on Digital libraries · doi:10.1145/336597.336644 · Cited by 1,147
Research works printed but not linked: 7
- Andrew M. Dai and Quoc V. Le. 2015. Semisupervised sequence learning. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada , pages 3079-3087.
- Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A diversitypromoting objective function for neural conversation models. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016 , pages 110-119.
- Gábor Melis, Chris Dyer, and Phil Blunsom. 2018. On the state of the art of evaluation in neural language models. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings .
- Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018. Regularizing and optimizing LSTM language models. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings .
- 1-6, 2018, Volume 1 (Long Papers) , pages 22272237.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog , 1(8).
- Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019b. What do you learn from context? Probing for sentence structure in contextualized word representations. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 .
Cited by
Items 501–600 of 1,040
Language Models with Rationality
Kassner, N., Tafjord, O., Sabharwal, A., Richardson, K., Schuetze, H., & Clark, P. A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.877 · Cited by 50
Emptying the Ocean with a Spoon: Should We Edit Models?
Pinter, Y., & Elhadad, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1012 · Cited by 34
What Else Do I Need to Know? The Effect of Background Information on Users’ Reliance on QA Systems
Goyal, N., Briakou, E., Liu, A., Baumler, C., Bonial, C., Micher, J. C., Voss, C. R., Carpuat, M. J., & Daumé, H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.201 · Cited by 19
Open-Ethical AI: Advancements in Open-Source Human-Centric Neural Language Models
Sicari, S., Moreno, J. F. C., Rizzardi, A., & Coen‐Porisini, A. (2024) · ACM Computing Surveys · doi:10.1145/3703454 · Cited by 17
MineGAN++: Mining Generative Models for Efficient Knowledge Transfer to Limited Data Domains
Wang, Y., González-García, A., Wu, C., Herranz, L., Khan, F. S., Jui, S., Yang, J., & van de Weijer, J. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01882-y · Cited by 13
A Few More Examples May Be Worth Billions of Parameters
Kirstain, Y., Lewis, P., Riedel, S., & Levy, O. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.72 · Cited by 35
CoCoLM: Complex Commonsense Enhanced Language Model with Discourse Relations
Yu, C., Zhang, H., Song, Y., & Ng, W. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.93 · Cited by 28
Jain, D., & Espinosa-Anke, L. (2022) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2022.starsem-1.13 · Cited by 21
KnowlyBERT - Hybrid Query Answering over Language Models and Knowledge Graphs
Kalo, J.-C., Fichtel, L., Ehler, P., & Balke, W. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-62419-4_17 · Cited by 11
Transferring Knowledge from Vision to Language: How to Achieve it and how to Measure it?
Norlund, T., Hagström, L., & Johansson, R. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.10 · Cited by 17
How to Query Language Models?Preprint
Adolphs, L., Dhuliawala, S., & Hofmann, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.01928 · Cited by 15
Integration AI Techniques in Low-Resource Language: The Case of Kazakh Language
Asmaganbetova, K., Auyeskhan, U., Zarina, N., Alen, S., & Yeskazina, M. (2024) · IEEE AITU: Digital Generation · doi:10.1109/ieeeconf61558.2024.10585350 · Cited by 13
Noise-Assisted Prompt Learning for Image Forgery Detection and Localization
Li, D., Zhu, J., Fu, X., Guo, X., Liu, Y., Yang, G., Liu, J., & Zha, Z.-J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73247-8_2 · Cited by 24
Zhuang, Y., Zhang, J., Li, X., Liu, C., Yu, Y., Dong, W., & He, K. (2024) · JMIR Medical Informatics · doi:10.2196/63020 · Cited by 10
KG-prompt: Interpretable knowledge graph prompt for pre-trained language models
Chen, L., Liu, J., Duan, Y., & Wang, R. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113118 · Cited by 12
Ma, Y., Qing, L., Kang, Y., Liu, J., Zhang, Y., Cheng, Q., Lu, W., & Liu, X. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127226 · Cited by 13
Zia, M. F., Ouameur, M. A., Bagaa, M., Massicotte, D., & Ksentini, A. (2025) · Physical Communication · doi:10.1016/j.phycom.2025.102857 · Cited by 9
Open-Domain Contextual Link Prediction and its Complementarity with Entailment Graphs
Hosseini, M. J., Cohen, S. B., Johnson, M., & Steedman, M. J. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.238 · Cited by 17
LM-CORE: Language Models with Contextually Relevant External Knowledge
Kaur, J. N., Bhatia, S., Aggarwal, M., Bansal, R., & Krishnamurthy, B. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.57 · Cited by 16
Exploring Pre-Trained Language Models to Build Knowledge Graph for Metal-Organic Frameworks (MOFs)
An, Y., Greenberg, J., Hu, X. T., Kalinowski, A., Fang, X., Zhao, X., McClellan, S., Uribe‐Romo, F. J., Langlois, K., Furst, J. D., et al. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10020568 · Cited by 16
Improving Pre-trained Language Models
Paaß, G., & Giesselbach, S. (2023) · Artificial intelligence: foundations, theory, and algorithms/Artificial intelligence: Foundations, theory, and algorithms · doi:10.1007/978-3-031-23190-2_3 · Cited by 10
Multilingual Knowledge Graph Completion from Pretrained Language Models with Knowledge Constraints
Song, R., He, S., Gao, S., Cai, L., Liu, K., Yu, Z., & Zhao, J. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.488 · Cited by 14
Beyond Positive Scaling: How Negation Impacts Scaling Trends of Language Models
Zhang, Y., Yasunaga, M., Zhou, Z., HaoChen, J. Z., Zou, J., Liang, P., & Yeung, S. Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.472 · Cited by 28
Robust Integration of Contextual Information for Cross-Target Stance Detection
Beck, T., Waldis, A., & Gurevych, I. (2023) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2023.starsem-1.43 · Cited by 10
Predictive Prompts with Joint Training of Large Language Models for Explainable Recommendation
Lin, C.-S., Tsai, C.-N., Su, S.-T., Jwo, J., Lee, C., & Wang, X. E. (2023) · Mathematics · doi:10.3390/math11204230 · Cited by 9
Wang, Y., Zhang, Y., Zhu, J., Liao, W., Yuan, M., & Zhou, W. G. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111129 · Cited by 10
Quantifying and Analyzing Entity-Level Memorization in Large Language Models
Zhou, Z., Xiang, J., Chen, C., & Su, S. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29948 · Cited by 37
A User-Friendly Framework for Generating Model-Preferred Prompts in Text-to-Image Synthesis
Hei, N., Guo, Q., Wang, Z., Wang, Y., Wang, H., & Zhang, W. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i3.27986 · Cited by 19
Ethical Considerations of Generative AI: A Survey Exploring the Role of Decision Makers in the Loop
Bautista, Y. J. P., Theran, C., & Aló, R. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v3i1.31243 · Cited by 15
A Communication Theory Perspective on Prompting Engineering Methods for Large Language Models
Song, Y., He, Y., Zhao, X.-F., Gu, H., Jiang, D., Yang, H., & Fan, L. (2024) · Journal of Computer Science and Technology · doi:10.1007/s11390-024-4058-8 · Cited by 14
Fine-grained multi-modal prompt learning for vision–language models
Liu, Y., Deng, Y., Liu, A., Liu, Y., & Li, S. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130028 · Cited by 10
Prompt-Based Multi-Interest Learning Method for Sequential Recommendation
Dong, X., Song, X., Liu, T., & Guan, W. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3563663 · Cited by 11
Soni, N., Matero, M., Balasubramanian, N., & Schwartz, H. A. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.52 · Cited by 31
Toward Prompt-Enhanced Sentiment Analysis with Mutual Describable Information Between Aspects
Xie, G., Liu, N., Hu, X., & Shen, Y. (2023) · Applied Artificial Intelligence · doi:10.1080/08839514.2023.2186432 · Cited by 10
Knowledge-Guided Prompt Learning for Few-Shot Text Classification
Wang, L., Chen, R., & Li, L. (2023) · Electronics · doi:10.3390/electronics12061486 · Cited by 12
Elaboration-Generating Commonsense Question Answering at Scale
Wang, W., Srikumar, V., Hajishirzi, H., & Smith, N. A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.90 · Cited by 16
FLamE: Few-shot Learning from Natural Language Explanations
Zhou, Y., Zhang, Y., & Tan, C. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.372 · Cited by 20
Interpreting Language Models Through Knowledge Graph ExtractionPreprint
Swamy, V., Romanou, A., & Jaggi, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.08546 · Cited by 15
EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Models
Wang, P., Zhang, N., Tian, B., Xi, Z., Yao, Y., Xu, Z., Wang, M., Mao, S., Wang, X., Cheng, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.07269 · Cited by 9
“The ability to edit and manipulate LLMs in a controlled and targeted manner may open up new possibilities for knowledge augmentation (Wu et al., 2023, 2020; Zhang et al., 2022; Chen et al., 2022) and adaptation across various natural language processing tasks (Kaddour et al., 2023).”
BackgroundCLadder: Assessing Causal Reasoning in Language Models
Jin, Z., Chen, Y., Leeb, F., Gresele, L., Kamal, O., Lyu, Z., Blin, K., Adauto, F. G., Kleiman-Weiner, M., Sachan, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.04350 · Cited by 8
Incorporating target-aware knowledge into prompt-tuning for few-shot stance detection
Wang, S., Sun, F., Wang, X., & Pan, L. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103815 · Cited by 10
Mutual Prompt Leaning for Vision Language Models
Long, S., Zhao, Z. W., Yuan, J., Tan, Z., Liu, J., Feng, J., Wang, S., & Wang, J. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02243-z · Cited by 11
Liao, W., Zhu, Y., Zhang, Z., Wang, Y., Wang, Z., Chu, X., Wang, Y., & Ma, L. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3690624.3709166 · Cited by 16
Synonyms, Antonyms and Factual Knowledge in BERT Heads
Serina, L., Putelli, L., Gerevini, A., & Serina, I. (2023) · Future Internet · doi:10.3390/fi15070230 · Cited by 10
TSGP: Two-Stage Generative Prompting for Unsupervised Commonsense Question Answering
Sun, Y., Zhang, Y., Qi, L., & Shi, Q. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.68 · Cited by 13
Selective-LAMA: Selective Prediction for Confidence-Aware Evaluation of Language Models
Yoshikawa, H., & Okazaki, N. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.150 · Cited by 38
Generating Variable Explanations via Zero-shot Prompt Learning
Wang, C., Lou, Y., Liu, J., & Peng, X. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00130 · Cited by 16
BioPRO: Context-Infused Prompt Learning for Biomedical Entity Linking
Zhu, T., Qin, Y., Feng, M., Chen, Q., Hu, B., & Xiang, Y. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3331149 · Cited by 12
ChatICD: Prompt Learning for Few-shot ICD Coding through ChatGPT
Liu, J., Yang, S., Peng, T., Hu, X., & Zhu, Q. (2023) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm58861.2023.10385482 · Cited by 9
He, G., Lin, C., Ren, J., & Duan, P. (2024) · Journal of Informetrics · doi:10.1016/j.joi.2024.101596 · Cited by 9
SE Factual Knowledge in Frozen Giant Code Model: A Study on FQN and Its Retrieval
Huang, Q. Y., Liao, D., Xing, Z., Yuan, Z., Lu, Q., Xu, X., & Lu, J. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3436883 · Cited by 15
M-Tuning: Prompt Tuning With Mitigated Label Bias in Open-Set Scenarios
Liao, N., Zhang, X., Cao, M., & Yan, J. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3524181 · Cited by 12
Wu, Q., Khan, M. A., Das, S., Nanda, V., Ghosh, B., Kolling, C., Speicher, T., Bindschaedler, L., Gummadi, K. P., & Terzi, E. (2025) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3701551.3703562 · Cited by 22
Transparency Helps Reveal When Language Models Learn Meaning
Wu, Z., Merrill, W., Peng, H. M., Beltagy, I., & Smith, N. A. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00565 · Cited by 19
ADEPT: Adapter-based Efficient Prompt Tuning Approach for Language Models
Shah, A., Thapa, S., Jain, A., & Huang, L. (2023) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2023.sustainlp-1.8 · Cited by 8
PromptDA: Label-guided Data Augmentation for Prompt-based Few Shot Learners
Chen, C., & Shu, K. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.41 · Cited by 15
Improving Probability-based Prompt Selection Through Unified Evaluation and Analysis
Yang, S., Kim, J., Jang, J., Ye, S., Lee, H., & Seo, M. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00666 · Cited by 19
Can Language Models Act as Knowledge Bases at Scale?
He, Q., Wang, Y., & Wang, W. (2024) · arXiv · doi:10.48550/arxiv.2402.14273 · Cited by 8
Generating benchmarks for factuality evaluation of language models
Muhlgay, D., Ram, O., Magar, I., Levine, Y., Ratner, N., Belinkov, Y., Abend, O., Leyton-Brown, K., Shashua, A., & Shoham, Y. (2023) · arXiv · doi:10.48550/arxiv.2307.06908 · Cited by 8
SCARE: A Novel Framework to Enhance Chinese Harmful Memes Detection
Gu, T., Feng, M., Feng, X., & Wang, X. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3481419 · Cited by 10
CoAPT: Context Attribute words for Prompt Tuning
Lee, G., An, S., Baik, S., & Lee, S. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113653 · Cited by 8
Ontology Population With Large Language Models (LLMs): A Case Study on Asbestos Ontology
Ayad, S., & Khelifa, L. C. (2025) · Applied Ontology · doi:10.1177/15705838251334528 · Cited by 10
Trappey, A. J. C., Wu, C.-Y., Lee, Y.-C., & Hung, L. P. (2025) · Journal of Engineering Design · doi:10.1080/09544828.2025.2518657 · Cited by 7
Fathallah, N., Bhole, M., & Staab, S. (2024) · International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion (DSAI) · doi:10.1145/3696593.3696617 · Cited by 8
An Automotive Fault Diagnosis Framework Based on Knowledge Graphs and Large Language Models
Lin, W., & Miao, K. (2025) · Electronics · doi:10.3390/electronics14214180 · Cited by 12
Pseudo Outlier Exposure for Out-of-Distribution Detection using Pretrained Transformers
Kim, J., Jung, K., Na, D., Jang, S., Park, E., & Choi, S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.95 · Cited by 15
Cross Modal Compression With Variable Rate Prompt
Gao, J., Li, J., Jia, C., Wang, S., Ma, S., & Gao, W. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3310657 · Cited by 10
Comparative study of typical neural solvers in solving math word problems
He, B., Yu, X., Huang, L., Meng, H., Liang, G., & Chen, S. (2024) · Complex & Intelligent Systems · doi:10.1007/s40747-024-01454-8 · Cited by 7
Cross-coupled prompt learning for few-shot image recognition
Zhang, F., Wei, R., Xie, Y., Wang, Y., Tan, X., Ma, L., Tang, M., & Fan, L. (2024) · Displays · doi:10.1016/j.displa.2024.102862 · Cited by 7
Unified Text-Image Space Alignment with Cross-Modal Prompting in CLIP for UDA
Jiao, Y., Cai, C., & Bao, B. (2025) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3715699 · Cited by 8
Aiyappa, R., Senthilmani, S., An, J., Kwak, H., & Ahn, Y. (2026) · PeerJ Computer Science · doi:10.7717/peerj-cs.3540 · Cited by 15
Long-term Control for Dialogue Generation: Methods and Evaluation
Ramakrishnan, R., Narangodage, H., Schilman, M., Weinberger, K. Q., & McDonald, R. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.54 · Cited by 8
An Exploration of Prompt-Based Zero-Shot Relation Extraction Method
Zhao, J., Hu, Y., Xu, N., Gui, T., Zhang, Q., Chen, Y., & Gao, X. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-18315-7_6 · Cited by 14
GraphPrompt: Graph-Based Prompt Templates for Biomedical Synonym Prediction
Xu, H., Zhang, J., Wang, Z., Zhang, S., Bhalerao, M. M., Liu, Y., Zhu, D., & Wang, S. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26256 · Cited by 10
Small Pre-trained Language Models Can be Fine-tuned as Large Models via Over-Parameterization
Gao, Z.-F., Zhou, K., Liu, P., Zhao, W. X., & Wen, J.-R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.212 · Cited by 25
Can Pretrained Language Models (Yet) Reason Deductively?
Yuan, Z., Hu, S., Vulić, I., Korhonen, A., & Meng, Z. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.106 · Cited by 15
Hypert: hypernymy-aware BERT with Hearst pattern exploitation for hypernym discovery
Yun, G., Lee, Y., Moon, A.-S., & Lee, J. (2023) · Journal Of Big Data · doi:10.1186/s40537-023-00818-0 · Cited by 7
Zhang, W., Wang, Y., & Fan, M. Y. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44192-9_15 · Cited by 10
Analyzing Semantic Faithfulness of Language Models via Input Intervention on Question Answering
Chaturvedi, A., Bhar, S., Saha, S., Garain, U., & Asher, N. M. (2023) · Computational Linguistics · doi:10.1162/coli_a_00493 · Cited by 16
TSP-Transformer: Task-Specific Prompts Boosted Transformer for Holistic Scene Understanding
Wang, S., Li, J., Zhao, Z., Lian, D., Huang, B., Wang, X., Li, Z., & Gao, S. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00097 · Cited by 16
Learning General Policies for Planning through GPT Models
Rossetti, N., Tummolo, M., Gerevini, A., Putelli, L., Serina, I., Chiari, M., & Olivato, M. (2024) · Proceedings of the International Conference on Automated Planning and Scheduling · doi:10.1609/icaps.v34i1.31510 · Cited by 42
Probing Pre-Trained Language Models for Disease Knowledge
Alghanmi, I., Espinosa-Anke, L., & Schockaert, S. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.266 · Cited by 13
Distantly Supervised Course Concept Extraction in MOOCs with Academic Discipline
Lu, M., Wang, Y., Yu, J., Du, Y., Hou, L., & Li, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.729 · Cited by 8
Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Views
Margatina, K., Wang, S., Vyas, Y., John, N. A., Benajiba, Y., & Ballesteros, M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.211 · Cited by 27
Eliciting Knowledge from Language Models for Event Extraction.Preprint
Lin, J., Jin, J., & Chen, Q. (2021) · arXiv (Cornell University) · Cited by 7
Animal-CLIP: A Dual-Prompt Enhanced Vision-Language Model for Animal Action Recognition
Jing, Y., Liang, K., Zhang, R., Sun, H., Li, Y., He, Z., & Ma, Z. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02408-4 · Cited by 12
scKGBERT: a knowledge-enhanced foundation model for single-cell transcriptomics
Li, Y. E., Qiao, G., Du, H., Gao, X., & Wang, G. (2025) · Genome biology · doi:10.1186/s13059-025-03862-6 · Cited by 6
Language Models Are Poor Learners of Directional Inference
Li, T., Hosseini, M. J., Weber, S., & Steedman, M. J. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.64 · Cited by 12
PALT: Parameter-Lite Transfer of Language Models for Knowledge Graph Completion
Shen, J., Wang, C., Yuan, Y., Han, J., Ji, H., Sen, K., Zhang, M., & Song, D. X. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.281 · Cited by 8
Category-Specific Prompts for Animal Action Recognition with Pretrained Vision-Language Models
Jing, Y., Wang, C., Zhang, R., Liang, K., & Ma, Z. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612551 · Cited by 10
ClusterPrompt: Cluster Semantic Enhanced Prompt Learning for New Intent Discovery
Liang, J., & Liao, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.702 · Cited by 24
A Knowledge-enhanced Two-stage Generative Framework for Medical Dialogue Information Extraction
Hu, Z., Ni, Z., Shi, J. Y., Xu, S., & Xu, B. (2024) · Machine Intelligence Research · doi:10.1007/s11633-023-1461-5 · Cited by 6
Augmenting Low-Resource Cross-Lingual Summarization with Progression-Grounded Training and Prompting
Ma, J., Huang, Y., Wang, L., Huang, X., Peng, H., Yu, Z., & Yu, P. S. (2024) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3675167 · Cited by 11
Implicit and explicit commonsense for multi-sentence video captioning
Chou, S.-H., Little, J. J., & Sigal, L. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.104064 · Cited by 6
Dual Prompt-Based Few-Shot Learning for Automated Vulnerability Patch Localization
Zhang, J., Hu, X., Bao, L., Xia, X., & Li, S. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00102 · Cited by 15
Action-guided prompt tuning for video grounding
Wang, J., Tsao, R., Wang, X., Wang, X., Feng, F., Tian, S., & Poria, S. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102577 · Cited by 6
Pouramini, A., & Faili, H. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05660-2 · Cited by 11
Wu, L., Zhao, Y., Hou, X., Liu, T., & Wang, H. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3645069 · Cited by 11
Morph Call: Probing Morphosyntactic Content of Multilingual Transformers
Mikhailov, V., Serikov, O., & Artemova, E. (2021) · doi:10.18653/v1/2021.sigtyp-1.10 · Cited by 13
CoHOZ: Contrastive Multimodal Prompt Tuning for Hierarchical Open-set Zero-shot Recognition
Liao, N., Liu, Y., Li, X., Lei, C., Wang, G., Hua, X., & Yan, J. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548021 · Cited by 9
Metadata
- Authors
- Jiang, Z., Xu, F. F., Araki, J., & Neubig, G.
- Affiliations
- Carnegie Mellon University (US)
- Robert Bosch (United States) (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Pages
- 423-438
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2991382858
- Cited by
- 1,002
- References
- 52
- Citations by year
- 2026: 76 · 2025: 169 · 2024: 224 · 2023: 285 · 2022: 144 · 2021: 78 · 2020: 25
- FWCI
- 52.35
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1815287)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- prompt generation, ensemble methods
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.