How Can We Know What Language Models Know?
Open accessThe DOI matches its Crossref record; 51 of the 52 references it cites were found in a registry or an index.
Abstract
Recent work has presented intriguing results examining the knowledge contained in language models (LMs) by having the LM fill in the blanks of prompts such as “ Obama is a __ by profession”. These prompts are usually manually created, and quite possibly sub-optimal; another prompt such as “ Obama worked as a __ ” may result in more accurately predicting the correct profession. Because of this, given an inappropriate prompt, we might fail to retrieve facts that the LM does know, and thus any given prompt only provides a lower bound estimate of the knowledge contained in an LM. In this paper, we attempt to more accurately estimate the knowledge contained in LMs by automatically discovering better prompts to use in this querying process. Specifically, we propose mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different prompts. Extensive experiments on the LAMA benchmark for extracting relational knowledge from LMs demonstrate that our methods can improve accuracy from 31.1% to 39.6%, providing a tighter lower bound on what LMs know. We have released the code and the resulting LM Prompt And Query Archive (LPAQA) at https://github.com/jzbjyb/LPAQA .
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The ability of Large Language Models to retrieve factual knowledge as explicit knowledge bases [53-56, 56-61] has been stirred discussed, which presents an opportunity to construct more comprehensive knowledge graphs within recommender systems.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [56]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“even experienced prompt designers may fail to manually discover optimal prompts [52].”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [52]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Beyond this, we found models trained with Toolformer to often be sensitive to the exact wording of their input when deciding whether or not to call an API; this is perhaps unsurprising given that LMs are known to be very sensitive to the prompt they are provided with in both zeroand few-shot settings (Jiang et al., 2020;Schick and Schütze, 2021a).”
From Toolformer: Language Models Can Teach Themselves to Use Tools (2023) · cites this paper as Jiang et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“There is some evidence to suggest that formats that occur commonly in the training data will lead to better performance (Jiang et al., 2020).”
From The Prompt Report: A Systematic Survey of Prompt Engineering Techniques (2024) · cites this paper as Jiang et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Jiang et al. [52]'s Mine approach is a mining-based method to automatically find templates given a set of training inputs x and outputs y.”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [52]
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“Prompts where the hidden vectors learned through a language modeling objective are then used in downstream language understanding systems (Dai and Le, 2015; Melamud et al., 2016; Peters et al., 2018; Devlin et al., 2019).”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
A Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1014 · Cited by 1,926
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834
ERNIE: Enhanced Language Representation with Informative Entities
Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., & Liu, Q. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1139 · Cited by 1,482 · cited 5×
“Similar extensions have been applied to pre-trained LMs like BERT, where contextualized representations are enhancedwith entity embeddings (Zhang et al., 2019; Peters et al., 2019; Pörner et al., 2019).”
MethodInfluentialOpen information extraction from the web
Banko, M., Cafarella, M., Soderland, S. G., Broadhead, M., & Etzioni, O. (2007) · International Joint Conference on Artificial Intelligence · Cited by 1,398
BERT Rediscovers the Classical NLP Pipeline
Tenney, I., Das, D., & Pavlick, E. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1452 · Cited by 1,307
(2019) · doi:10.18653/v1/p19-1356 · Cited by 1,291 · cited 1×
“…a broad suite of NLP tasks are used to analyze both syntactic and semantic properties, providing evidence that contextualized representations encode linguistic knowledge in different layers (Hewitt and Manning, 2019; Tenney et al., 2019a; Tenney et al., 2019b; Jawahar et al., 2019; Goldberg, 2019).”
BackgroundSnowball: extracting relations from large plain-text collections
Agichtein, E., & Gravano, L. (2000) · Proceedings of the fifth ACM conference on Digital libraries · doi:10.1145/336597.336644 · Cited by 1,147
Research works printed but not linked: 7
- Andrew M. Dai and Quoc V. Le. 2015. Semisupervised sequence learning. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada , pages 3079-3087.
- Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A diversitypromoting objective function for neural conversation models. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016 , pages 110-119.
- Gábor Melis, Chris Dyer, and Phil Blunsom. 2018. On the state of the art of evaluation in neural language models. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings .
- Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018. Regularizing and optimizing LSTM language models. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings .
- 1-6, 2018, Volume 1 (Long Papers) , pages 22272237.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog , 1(8).
- Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019b. What do you learn from context? Probing for sentence structure in contextualized word representations. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 .
Cited by
Items 901–1,000 of 1,040
SIFT-HF Scale-Invariant Feature-based similarity Threshold from Human FeedbackPreprint
Cyrusian, S. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5241660 · Cited by 1
Evaluation and Integration of Structural Knowledge for Pre-trained Language Models
Lovón-Melgarejo, J. (2024) · theses.fr (ABES) · doi:10.70675/9b2d6bf6za13az4dfcza45bzca75ad5b25a6 · Cited by 2
Towards Real World Simulation by Natural Language
Kobayashi, I. (2021) · Journal of the Robotics Society of Japan · doi:10.7210/jrsj.39.391
Benchmarking Commonsense Knowledge Base Population with an Effective Evaluation DatasetPreprint
Fang, T., Wang, W., Choi, S.-H., Hao, S., Zhang, H., Song, Y., & He, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.07679 · Cited by 2
Pathologies of Pre-trained Language Models in Few-shot Fine-tuning
Chen, H., Zheng, G., Awadallah, A. H., & Ji, Y. (2022) · Workshop on Insights from Negative Results in NLP (Insights) · doi:10.18653/v1/2022.insights-1.20 · Cited by 3
Prompt as a Knowledge Probe for Chinese Spelling Check
Peng, K., Sun, N., Cao, J., Liu, R., Ren, J., & Jiang, L. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-10989-8_41
Few-Shot Learning for Medical Numerical Understanding Based on Machine Reading Comprehension
Zeng, X., Hu, W., Liu, X., Chen, Y., Shao, W., & Sun, L. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28124-2_58
Multihop Factual Claim Verification Using Natural Language Prompts
Rahman, M. M., Islam, M. S., Laskar, M. T. R., Hossain, M. A., & Kamal, A. R. M. (2023) · doi:10.21428/594757db.63abb0f0
Zaninello, A., & Magnini, B. (2023) · NLRSE · doi:10.18653/v1/2023.nlrse-1.3
Can LMs Store and Retrieve 1-to-N Relational Knowledge?
Nagasawa, H., Heinzerling, B., Kokuta, K., & Inui, K. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-srw.22
DynaMaR: Dynamic Prompt with Mask Token Representation
Sun, X., Rajagopalan, S., Nigam, P., Lu, W., Xu, Y., Keivanloo, I., Zeng, B., & Chilimbi, T. M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-industry.2
Controlling for Stereotypes in Multimodal Language Model Evaluation
Malik, M., & Johansson, R. (2022) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2022.blackboxnlp-1.21
Simple Knowledge Graph Completion Model Based on Differential Negative Sampling and Prompt Learning
Duan, L., Wang, J., Luo, B., & Sun, Q. (2023) · Information · doi:10.3390/info14080450 · Cited by 1
Probing Out-of-Distribution Robustness of Language Models with Parameter-Efficient Transfer Learning
Cho, H., Park, C., Kim, J., Kim, H. J., Yoo, K. M., & Lee, S. (2023) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2023.starsem-1.21 · Cited by 2
Region-Attention Prompt Learning for CLIP
Pan, Y., Cheng, H. N., Fang, Y., & Liu, Y. (2023) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-230879
Pre-trained Models for Representation Learning
Lin, Y., Ding, N., Liu, Z., & Sun, M. (2023) · Springer eBooks · doi:10.1007/978-981-99-1600-9_5 · Cited by 2
Prediction of Sumoylation Sites in Proteins From Language Model Representations
Sidorov, E. (2023) · Digital Commons - Michigan Tech (Michigan Technological University) · doi:10.37099/mtu.dc.etdr/1607
Vulnerability Analysis of Continuous Prompts for Pre-trained Language Models
Li, Z., Shi, Y., Sheng, X., Yin, C., Zhou, L. Q., & Li, P. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44201-8_41
Instance-Wise Adaptive Tuning and Caching for Vision-Language Models
Yang, C., Meng, F., Chen, S., Liu, M., & Zhang, R. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230595
TALCLIP: CLIP for Temporal Action Localization
Wang, K., & Yu, B. (2023) · doi:10.1109/ictech58362.2023.00083
Multi-angle Prediction Based on Prompt Learning for Text Classification
Ju, Z., Li, Z., Wu, S., Zhao, X., & Zhan, Y. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44699-3_25
CKKP: Chinese-Korean Text Classification via Knowledge Base and Prompt Learning
Wen, H., Zhang, Z., & Lu, X. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-7869-4_33
Hai-Jew, S. (2023) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-0074-9.ch005
Learning to Predict Task Transferability via Soft Prompt
Feng, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.546 · Cited by 3
AutoReply: Detecting Nonsense in Dialogue with Discriminative Replies
Shi, W., Dinan, E., Renduchintala, A., Fried, D., Jacob, A. P., Zhou, Y., & Lewis, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.23 · Cited by 1
Test-time Augmentation for Factual Probing
Kamoda, G., Heinzerling, B., Sakaguchi, K., & Inui, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.236 · Cited by 1
Simple and Effective Input Reformulations for Translation
Yu, B., Lillemark, H. J., & Keutzer, K. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.638
Qi, J. (2024) · Applied and Computational Engineering · doi:10.54254/2755-2721/30/20230096
A New Model for the Arabic Language Based on the Semantic Representation of Words and Sentences
Awajan, A. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-47718-8_3
Text Prompt Translation for Zero Shot Prediction under Lexical Constraint in Aviation Domain
Chen, F., Fang, Y., Man, X., Zhao, Y., Fang, T., Lin, L., Tan, M., & Huang, S. (2023) · International Conference on Software Engineering and Computer Science (CSECS) · doi:10.1109/csecs60003.2023.10428175
Conversational Recommender Systems based on Topic Prediction and Retrieval
Cui, X. F., & Li, L. (2023) · International Conference on Information Education and Artificial Intelligence (ICIEAI) · doi:10.1145/3660043.3660126 · Cited by 1
WiP: Towards Light Adaptation of Large Language Models For Personal Hardware
Wang, L., Wang, J., & Wang, D. (2024) · EdgeFM@MobiSys · doi:10.1145/3662006.3662065
Multi-Relation Extraction for Cybersecurity Based on Ontology Rule-Enhanced Prompt Learning
Wang, F., Ding, Z., Liu, K., Xin, L., Zhao, Y., & Zhou, Y. (2024) · Electronics · doi:10.3390/electronics13122379
Benchmarking Biomedical Relation Knowledge in Large Language Models
Zhang, F., Yang, K., Zhao, C., Li, H., Xin, D., Tian, H., & Zhou, X. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5131-0_41
Enhancing Data Augmentation with Knowledge-enriched Data Generation via Dynamic Prompt-tuning Method
Qi, Q., Bao, Q., Peng, A. Y., Liu, J., & Witbrock, M. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651072
Option-Differentiated Clue Augmentation for Commonsense Question Answering
Li, H., Deng, W., Wang, Z., Wang, K., Zhuang, Z., & Yang, D. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650381
PLSRP: prompt learning for send–receive path prediction
Wei, Q., Yi-hong, L., Du, D., Cai, H.-T., Lai, P.-Y., & Wang, C. (2024) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02387-x
Sedkaoui, S. (2024) · Wiley eBooks · doi:10.1002/9781394325498.refs
Learning Cross-modal Knowledge Reasoning and Heuristic-prompt for Visual-language Navigation
Zhou, D., Pang, Z., & Li, W. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679740 · Cited by 1
Assessing the Syllogistic Logic and Fact-Checking Capabilities of Large Language Models
Solorzano, C. D., Delaflor, M., & Toxtli, C. (2024) · IEEE International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData) · doi:10.1109/ithings-greencom-cpscom-smartdata-cybermatics62450.2024.00094
Prompt-Based Joint Contrastive Learning for Zero-Shot Relation Extraction
Zou, J., Xiao, Y., Zhou, S., Li, W., & Yang, Q. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9431-7_32
CKF: Conditional Knowledge Fusion Method for CommonSense Question Answering
Xie, M., Hao, C., Zhang, P., & Ma, X. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9431-7_9
LMATDet:Lightweight Multi-scale Aerial TargetDetection with attention and efficient lossPreprint
Zhao, Y. (2024) · Research Square · doi:10.21203/rs.3.rs-5335228/v1
Evaluating the Performance to Detecting Topic Shift of Large Language Models: A Study
Tian, P., & Liu, D. (2024) · International Conference on Intelligent Computing and Signal Processing (ICSP) · doi:10.1109/icsp62122.2024.10743815
PSSOP: A Prompt-style Approach for Few-Shot Text Classification
Wang, Y., & Lv, C. (2024) · International Conference on Computational Linguistics and Natural Language Processing (CLNLP) · doi:10.1109/clnlp64123.2024.00029
Does ChatGPT have sociolinguistic competence?
Duncan, D. (2024) · Journal of Computer-Assisted Linguistic Research · doi:10.4995/jclr.2024.21958 · Cited by 7
One-Shot Classification Is Enough for Automatic Label Mapping
Lin, X., Aysa, A., & Ubul, K. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78169-8_25
TuneVLSeg: Prompt Tuning Benchmark for Vision-Language Segmentation Models
Adhikari, R., Thapaliya, S., Dhakal, M., & Khanal, B. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0908-6_3 · Cited by 1
Alleviating the medical strain: a triage method via cross-domain text classification
Xiao, X., Wang, S., Jiang, F., Qi, T., & Wang, W. (2024) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2024.1468519
Development of a Language Model to Enhance the Performance of Hindi Automatic Speech Recognition
Kumar, M. J. D., & Niranjan, P. (2024) · International Conference on Electronics, Communication and Aerospace Technology (ICECA) · doi:10.1109/iceca63461.2024.10801144
Disentangled Prompt Learning for Transferable, Multimodal, Few-Shot Image Classification
Yang, J. J., Magnani, A., & Yang, B. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825383
Decrease the Prompt Uncertainty: Adversarial Prompt Learning for Few-shot Text Classification
Weng, J., Zhang, Z., Ya-Qi, J., Niu, C., Huang, H., & Hu, Y. (2024) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc54092.2024.10831613
SSP: A Simple and Safe Prompt Engineering for Visual Generation
Cheng, W., Liu, J., Jiao, Z., Deng, J., & Ren, F. (2024) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc54092.2024.10832083 · Cited by 1
Knowledge Neurons in the Knowledge Graph-based Link Prediction Models
Mika, G. P., Bouzeghoub, A., Węgrzyn-Wolska, K. M., & Neggaz, Y. M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-82697-9_14
Building Proficiency in Generative AI: Key Competencies for SuccessPreprint
Grimberg, E., & Mason, C. (2025) · Qeios · doi:10.32388/infmmj.2
Li, X., Wu, X., Xiao, L., Liu, T., Huang, P. S., & Shan, J. (2026) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde65706.2026.00355
Generative Adversarial Network with Structured Semantic Prompts Constrainting Clip for Text-to-Image
Ge, S., Zhang, L., Xing, H., & Wu, X. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889641
Unlocking Proficiency: Experts’ Views on the Use of Generative AIPreprint
Grimberg, E., & Mason, C. (2025) · Qeios · doi:10.32388/infmmj.3
Utilizing Large Language Models to Detect and Decipher Abbreviation in Clinical Notes (Preprint)Preprint
Jafarpour, H., Cheligeer, C., Yan, J., Xu, Y., Eastwood, C. A., Southern, D. A., Zeng, Y., & Quan, H. (2025) · doi:10.2196/preprints.74042
Liu, J., & Liu, C. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-4298-4_16 · Cited by 1
Prompt Tuning Techniques for Chinese Idiom Recommendation
Wang, S.-M., Su, I., & Chung, Y. (2025) · IEEE Access · doi:10.1109/access.2025.3561188
Personalized recommendation with clustering via prompt-tuning
Wu, T., Liu, X., Zhu, Y. Q., Li, Y., Yuan, Y., & Qiang, J. (2025) · Intelligent Data Analysis · doi:10.1177/1088467x251331821 · Cited by 1
Search-Based Negative Prompt Optimisation for Text-to-Image Generation
Iglesias, G., Zamorano, M., & Sarro, F. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-90167-6_7 · Cited by 1
Explainable and Efficient Editing for Large Language Models
Zhang, T., Fang, J., Jiang, H., Bi, B., Wang, X., & He, X. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714835 · Cited by 10
Conditional Visuo-Textual Prompt Learning for Medical Image Analysis
Mineo, R., Bellitto, G., Salanitri, F. P., Hendrix, R., Bağcı, U., Giordano, D., Spampinato, C., & Palazzo, S. (2025) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi60581.2025.10980870
AI-Enhanced Decision Support for Optimized Carbon Quota Allocation Across Industrial Sectors
Liu, B., Liu, W., & Zhao, J. (2024) · IEEE Conference on Energy Internet and Energy System Integration (EI2) · doi:10.1109/ei264398.2024.10990380
Structure-Enhanced Prompt Learning for Graph-Based Code Vulnerability Detection
Chang, W., Ye, C., & Zhou, H. (2025) · Applied Sciences · doi:10.3390/app15116128 · Cited by 1
Negative prompt-guided optimization: Enhancing soft prompt generalization in vision-language models
Kim, S., & Lee, S. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.109093
Can Dynamic Prompt Help Sentiment Style Transfer?
Xu, S., Fukumoto, F., Go, K., & Suzuki, Y. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-7030-7_19
Design of an Online English Oral Examination System Based on Particle Filtering Algorithm
Du, L. (2025) · International Journal of High Speed Electronics and Systems · doi:10.1142/s0129156425407466
Asset-Centric Factor-Opinion Pair Extraction from Market Analysis Reports
Liu, S., & Wang, X. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-9914-8_26
Do Not Say It Directly: Generating Indirect Expressions with Large Language Models
Yu, P., Mei, T., Yang, D., & Liang, J. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-0020-8_38
Case studies in prompt engineering
Nayyar, A., Vairamani, A. D., & Kaswan, K. S. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-33904-2.00006-9
Poly-TF: A polymeric transformer framework for multiple visual tasks at once
Fan, X., Zhang, C., Sun, T., Gao, J., Xie, T., Dai, K., Zhao, L.-J., & Li, R. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114167
MALAMUTE: A Multilingual, Highly-granular, Template-free, Education-based Probing Dataset
Shaier, S., Baker, G. A., Sridhar, C., Hunter, L., & von der Wense, K. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.209 · Cited by 4
Prompt Tuning for Vision-Language Models via Meta-Training
Kim, D., & Baik, S. (2025) · Journal of Broadcast Engineering · doi:10.5909/jbe.2025.30.4.571
Li, Z., Lian, D., Peng, X., Huang, W., Zeng, X., Zhu, D. L., & Huang, G. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-7036-9_19
N2C2: Nearest Neighbor Enhanced Confidence Calibration for Cross-Lingual In-Context Learning
He, J., Yu, S., Gutiérrez-Basulto, V., & Pan, J. Z. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.07.121
Research on Data Construction Method of Entity Relation Instruction for Biological Invasion
Wu, Q., Liu, J., Fu, M.-Y., Yan, Y., & Wang, L. (2025) · International Symposium on Bioinformatics and Computational Biology (BCB) · doi:10.1145/3747357.3747387
Learning Efficient Feature Adapters for Vision-Language Models
Zhang, R., & Gao, P. (2025) · Advances in computer vision and pattern recognition · doi:10.1007/978-3-031-94969-2_7
Deng, Z., Wang, R., Xue, L., & Yang, J. (2025) · Displays · doi:10.1016/j.displa.2025.103217
Enhancing Aspect-Based Sentiment Analysis Through Contextual Text Generation and Data Augmentation
Kesar, M., Kumar, V., Singh, H., & Gupta, M. (2025) · Artificial intelligence-enhanced software and systems engineering · doi:10.1007/978-3-031-99882-9_16
Knowledge-Aware Prompt-Tuning for Integrated Conversational Recommender System
Yang, X.-H., Liu, M.-W., Luo, H., Zhou, S.-X., Ma, G., & Jiang, P. (2025) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2025.3608280
Keyan, J. I. N., Zhong, J., & Xu, L. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5624245
Kashlan, R., Duggireddy, H., Smith, J. J., Faraj, R. R., Gattas, S., Grossberg, J. A., & Gichoya, J. W. (2025) · Research Square · doi:10.21203/rs.3.rs-7744596/v1 · Cited by 1
Technical Contradiction Extraction Based on Two-Stage Knowledge Fusion Mechanism
Sun, W., Zhao, Y., Xi, C., & Lyu, L. (2025) · IFIP advances in information and communication technology · doi:10.1007/978-3-032-08851-2_20
The rhetorical move–step structure of highly cited articles in linguistics
Harfiani, M., Kurniawan, E., & Dallyono, R. (2025) · JOALL (Journal of Applied Linguistics and Literature) · doi:10.33369/joall.v10i2.42267
Ni, X., Zhao, Q., & Huang, S. (2025) · Scientia Sinica Informationis · doi:10.1360/ssi-2024-0329
Chen, Z., Li, D., Zhou, J., Wu, S., Ye, H., Lou, J., & Ng, S.-K. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3760803
Soft Prompt-Tuning for Personalized News Recommendation
Yuan, X., Geng, Y., Zhu, Y. Q., Zhao, Z., & Zhang, R. (2025) · The European Journal on Artificial Intelligence · doi:10.1177/30504554251388589
Research on Judgment Questions Based on Label Guided Prompt Generation and Adapter Mapping
Peng, X., & Zhang, Y. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228332
Similarity-based Prompt Optimization for Distilling Factual Knowledge From Language Models
Li, N., Liu, H., Peng, D., & Bouraoui, Z. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228874
Song, S., Wu, L., & Wang, L. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11229141
Jia, Y., Editorial Office, Zhang, Y., & Gao, Y. (2025) · Journal of Advanced Computational Intelligence and Intelligent Informatics · doi:10.20965/jaciii.2025.p1249
MetaPipe: Predicting Metaphoric Associations and Turning Your Metaphoric Imagination into Reality
Qian, W., Hu, Z., Hu, J., Li, J., & Hong, R. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2581257
Knowledge Editing of Large Language Models Unconstrained by Word Order
Ishigaki, R., Suzuki, J., Shuzo, M., & Maeda, E. (2025) · Journal of Natural Language Processing · doi:10.5715/jnlp.32.1062
Aftab, W., Apostolou, Z., Bouazoune, K., & Straub, T. (2024) · Figshare · doi:10.6084/m9.figshare.26860661
Exploring Interpretability of Knowledge Graph-enhanced Large Language Models
Tiwari, S. (2023) · Open Research Knowledge Graph (ORKG) · doi:10.48366/r601697
Metadata
- Authors
- Jiang, Z., Xu, F. F., Araki, J., & Neubig, G.
- Affiliations
- Carnegie Mellon University (US)
- Robert Bosch (United States) (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Pages
- 423-438
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2991382858
- Cited by
- 1,002
- References
- 52
- Citations by year
- 2026: 76 · 2025: 169 · 2024: 224 · 2023: 285 · 2022: 144 · 2021: 78 · 2020: 25
- FWCI
- 52.35
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1815287)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- prompt generation, ensemble methods
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.