How Can We Know What Language Models Know?
Open accessThe DOI matches its Crossref record; 51 of the 52 references it cites were found in a registry or an index.
Abstract
Recent work has presented intriguing results examining the knowledge contained in language models (LMs) by having the LM fill in the blanks of prompts such as “ Obama is a __ by profession”. These prompts are usually manually created, and quite possibly sub-optimal; another prompt such as “ Obama worked as a __ ” may result in more accurately predicting the correct profession. Because of this, given an inappropriate prompt, we might fail to retrieve facts that the LM does know, and thus any given prompt only provides a lower bound estimate of the knowledge contained in an LM. In this paper, we attempt to more accurately estimate the knowledge contained in LMs by automatically discovering better prompts to use in this querying process. Specifically, we propose mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different prompts. Extensive experiments on the LAMA benchmark for extracting relational knowledge from LMs demonstrate that our methods can improve accuracy from 31.1% to 39.6%, providing a tighter lower bound on what LMs know. We have released the code and the resulting LM Prompt And Query Archive (LPAQA) at https://github.com/jzbjyb/LPAQA .
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The ability of Large Language Models to retrieve factual knowledge as explicit knowledge bases [53-56, 56-61] has been stirred discussed, which presents an opportunity to construct more comprehensive knowledge graphs within recommender systems.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [56]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“even experienced prompt designers may fail to manually discover optimal prompts [52].”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [52]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Beyond this, we found models trained with Toolformer to often be sensitive to the exact wording of their input when deciding whether or not to call an API; this is perhaps unsurprising given that LMs are known to be very sensitive to the prompt they are provided with in both zeroand few-shot settings (Jiang et al., 2020;Schick and Schütze, 2021a).”
From Toolformer: Language Models Can Teach Themselves to Use Tools (2023) · cites this paper as Jiang et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“There is some evidence to suggest that formats that occur commonly in the training data will lead to better performance (Jiang et al., 2020).”
From The Prompt Report: A Systematic Survey of Prompt Engineering Techniques (2024) · cites this paper as Jiang et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Jiang et al. [52]'s Mine approach is a mining-based method to automatically find templates given a set of training inputs x and outputs y.”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [52]
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“Prompts where the hidden vectors learned through a language modeling objective are then used in downstream language understanding systems (Dai and Le, 2015; Melamud et al., 2016; Peters et al., 2018; Devlin et al., 2019).”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
A Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1014 · Cited by 1,926
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834
ERNIE: Enhanced Language Representation with Informative Entities
Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., & Liu, Q. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1139 · Cited by 1,482 · cited 5×
“Similar extensions have been applied to pre-trained LMs like BERT, where contextualized representations are enhancedwith entity embeddings (Zhang et al., 2019; Peters et al., 2019; Pörner et al., 2019).”
MethodInfluentialOpen information extraction from the web
Banko, M., Cafarella, M., Soderland, S. G., Broadhead, M., & Etzioni, O. (2007) · International Joint Conference on Artificial Intelligence · Cited by 1,398
BERT Rediscovers the Classical NLP Pipeline
Tenney, I., Das, D., & Pavlick, E. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1452 · Cited by 1,307
(2019) · doi:10.18653/v1/p19-1356 · Cited by 1,291 · cited 1×
“…a broad suite of NLP tasks are used to analyze both syntactic and semantic properties, providing evidence that contextualized representations encode linguistic knowledge in different layers (Hewitt and Manning, 2019; Tenney et al., 2019a; Tenney et al., 2019b; Jawahar et al., 2019; Goldberg, 2019).”
BackgroundSnowball: extracting relations from large plain-text collections
Agichtein, E., & Gravano, L. (2000) · Proceedings of the fifth ACM conference on Digital libraries · doi:10.1145/336597.336644 · Cited by 1,147
Research works printed but not linked: 7
- Andrew M. Dai and Quoc V. Le. 2015. Semisupervised sequence learning. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada , pages 3079-3087.
- Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A diversitypromoting objective function for neural conversation models. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016 , pages 110-119.
- Gábor Melis, Chris Dyer, and Phil Blunsom. 2018. On the state of the art of evaluation in neural language models. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings .
- Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018. Regularizing and optimizing LSTM language models. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings .
- 1-6, 2018, Volume 1 (Long Papers) , pages 22272237.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog , 1(8).
- Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019b. What do you learn from context? Probing for sentence structure in contextualized word representations. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 .
Cited by
Items 801–900 of 1,040
PDbDa: Prompt-tuned dual-branch framework for unsupervised domain adaptation
Zhang, Q., Zhao, Y., Shao, M., & Liang, H. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114509 · Cited by 2
Forward-Only Continual Learning
Chen, J., He, J., Chen, F., Lv, Z., & Tang, J. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755302 · Cited by 2
Gross, R. D., Tzach, Y., Halevi, T., Koresh, E., & Kanter, I. (2025) · Physica A Statistical Mechanics and its Applications · doi:10.1016/j.physa.2025.131102 · Cited by 7
Minimally-Supervised Relation Induction from Pre-trained Language Model
Sun, L., Shen, Y., & Lü, W. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.135 · Cited by 5
Confli-T5: An AutoPrompt Pipeline for Conflict Related Text Augmentation
Parolin, E. S., Hu, Y., Khan, L. R., Brandt, P. T., Osorio, J., & D'Orazio, V. J. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10020509 · Cited by 9
Class Cardinality Comparison as a Fermi Problem
Ghosh, S., Razniewski, S., & Weikum, G. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543873.3587334 · Cited by 3
The Exploration of Knowledge-Preserving Prompts for Document Summarisation
Chen, C., Zhang, W. E., Shakeri, A. S., & Fiza, M. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191910 · Cited by 5
Attribute Controlled Dialogue Prompting
Liu, R., Rashid, A., Kobyzev, I., Rezagholizadeh, M., & Poupart, P. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.150 · Cited by 2
Probing Relational Knowledge in Language Models via Word Analogies
Rezaee, K., & Camacho-Collados, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.289 · Cited by 2
Flatness-Aware Prompt Selection Improves Accuracy and Sample Efficiency
Shen, L., Tan, W., Zheng, B., & Khashabi, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.523 · Cited by 5
Do Language Models Learn about Legal Entity Types during Pretraining?
Barale, C., Rovatsos, M., & Bhuta, N. (2023) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2023.nllp-1.4 · Cited by 3
ClozEx: A Task toward Generation of English Cloze Explanation
Zhang, Z., Mita, M., & Komachi, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.347 · Cited by 3
All Should Be Equal in the Eyes of LMs: Counterfactually Aware Fair Text Generation
Banerjee, P., Java, A., Jandial, S., Shahid, S., Furniturewala, S., Krishnamurthy, B., & Bhatia, S. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29719 · Cited by 10
Legal text basic element identification based on the BERT model in the judicial field
Li, X. (2024) · Journal of Computational Methods in Sciences and Engineering · doi:10.3233/jcm-247453 · Cited by 2
Reconsidering the Past: Optimizing Hidden States in Language Models
Yoshida, D., & Gimpel, K. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.346 · Cited by 5
What do Models Learn From Training on More Than Text? Measuring Visual Commonsense Knowledge
Hagström, L., & Johansson, R. (2022) · Annual Meeting of the Association for Computational Linguistics: Student Research Workshop (ACL SRW) · doi:10.18653/v1/2022.acl-srw.19 · Cited by 5
UoM&MMU at TSAR-2022 Shared Task: Prompt Learning for Lexical Simplification
Vásquez-Rodríguez, L., Nguyen, N. T. H., Shardlow, M., & Ananiadou, S. (2022) · Workshop on Text Simplification, Accessibility and Readability (TSAR) · doi:10.18653/v1/2022.tsar-1.23 · Cited by 2
Acquiring and modelling abstract commonsense knowledge via conceptualization
He, M. (2022) · doi:10.14711/thesis-991013098357503412 · Cited by 5
Zheng, J., Zheng, R., & Zang, A. Y. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4764707 · Cited by 2
Zhi, H., Qiu, Y., Jiang, H., Xiao, M., Han, L., Lei, S., & Niu, H. (2025) · Scientific Data · doi:10.1038/s41597-025-05268-2 · Cited by 3
Representation, information extraction, and summarization for automatic multimedia understanding
Harrando, I. (2022) · theses.fr (ABES) · doi:10.70675/702b8ba3z3acaz4ce3za9b9zf2cb1609e328 · Cited by 2
Metheniti, E. (2023) · theses.fr (ABES) · doi:10.70675/2f81071bz4724z44ecz999dz1acee85d8951 · Cited by 5
Do as We Do, Not as You Think: the Conformity of Large Language Models
Weng, Z., Chen, G., & Wang, W. (2025) · arXiv · doi:10.48550/arxiv.2501.13381 · Cited by 1
“Knowledge Parameterization and Conflicts in LLMs. LLMs demonstrate the ability to inter-nalize factual knowledge during pre-training, essentially forming their own implicit knowledge base [62, 63].”
P 3 L: Patent Prediction With Prompt Learning
Lu, Y.-H., Lai, P.-Y., Chen, M.-S., Cai, H.-T., Wang, Z.-H., Liu, S., Dai, Q., & Wang, C. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3635881 · Cited by 2
FocusPatch AD: Few-Shot Multi-Class Anomaly Detection With Unified Keywords Patch Prompts
Ding, X., Li, X., Chen, M., Gong, J., & Xie, Y. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3646861 · Cited by 3
Wang, C., Nie, W., Zhang, H., Sun, H., & Ye, M. (2026) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2026.3658987 · Cited by 1
A2Net: Affiliation Alignment Networks for Whole-Body Pose Estimation With Vision--Language Models
Lin, L., Wang, Y., Zhu, C., & Chen, J. (2026) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2026.3656293 · Cited by 1
Optimizing medical image analysis with prompt-guided pseudo-labeling
Fan, L., Gong, X., Zheng, C., Li, X., Li, J., & Ou, Y. (2026) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2026.109867 · Cited by 1
Stengel-Eskin, E., & Durme, B. V. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.760 · Cited by 5
A Morphological Analysis: A Trend of the English Suffix “-less” in Indonesian Words on X/Twitter
Septiana, A. A., Kirana, A. F. P., Mahadewi, R. I. U., & Sujatna, E. T. S. (2023) · Pubmedia Jurnal Pendidikan Bahasa Inggris · doi:10.47134/jpbi.v1i2.209 · Cited by 1
Review on ChatGPT: History, Applications and Limitations in Chemistry
Etim, E. I. (2023) · Physical Science & Biophysics Journal · doi:10.23880/psbj-16000263 · Cited by 1
General Phrase Debiaser: Debiasing Masked Language Models at a Multi-Token Level
Shi, B., Zhang, X., Kong, D., Wu, Y., Liu, Z., Lyu, H., & Huang, L. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447098 · Cited by 4
Gould, J. A., Stawicki, S. P., Lee, R. Y., & Ng-Pellegrino, A. (2024) · Artificial intelligence · doi:10.5772/intechopen.115397 · Cited by 3
Building Natural Language Interface for Product Search
Malik, V., Puranik, V. S., Majumder, A., & Sembium, V. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3680070 · Cited by 3
Training-Free Feature Reconstruction with Sparse Optimization for Vision-Language Models
Zhang, Y. H., Yu, K., Avilés-Rivero, A. I., Jia, J., Tang, Y., & He, Z. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3680710 · Cited by 1
Prompt Engineering 101 Prompt Engineering Guidelines from a Linguistic Perspective
Han, W., Wei, X., Cui, X., Cheng, N., Jiang, G., Qian, W., & Zhang, C. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-8367-0_34 · Cited by 4
Constraint embedding for prompt tuning in vision-language pre-trained model
Cheng, K., Wei, L., Tang, J., & Zhan, Y. (2024) · Multimedia Systems · doi:10.1007/s00530-024-01600-9 · Cited by 1
Enhancing TrUStAPIS Methodology in the Web of Things with LLM-Generated IoT Trust Semantics
Ferraris, D., Kotis, K., & Kalloniatis, C. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-8798-2_7 · Cited by 1
Vo, T., Luu, S. T., & Nguyen, L.-M. (2025) · Knowledge and Information Systems · doi:10.1007/s10115-025-02345-1 · Cited by 3
ChatGPT and BERT: Comparative Analysis of Various Natural Language Processing Applications
Saranya, M., & Amutha, B. (2025) · Springer eBooks · doi:10.1007/978-981-96-2355-6_2 · Cited by 1
Adpl: attentive dual-modality prompt learning for vision-language understanding
Deng, Z., Wang, R., Xue, L., & Yang, J. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-07392-7 · Cited by 1
Yoo, M. H., & Song, S. (2025) · Lingua · doi:10.1016/j.lingua.2025.103998 · Cited by 1
Financial Inclusion and Large Language Models
Ozili, P. K., Obiora, K. I., & Onuzo, C. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-1250-7.ch010 · Cited by 1
Yang, Y., Li, C., Han, Z., Li, R., Xu, K., Li, Q., Zhong, W., Shen, Z., Fei, Z., Ge, J., et al. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3764584 · Cited by 12
Multimodal prompt learning with selective feature fusion: towards robust cross-modal alignment
Han, J., Wang, Y., Zhong, W., Zhang, Y., & Yuan, X. (2025) · Applied Intelligence · doi:10.1007/s10489-025-06919-y · Cited by 1
Learning Joint Text and Visual Tokens in CLIP for Medical Image Analysis
Mineo, R., Bellitto, G., Salanitri, F. P., Hendrix, R., Spampinato, C., & Palazzo, S. (2025) · International Workshop on Multimedia Computing for Health and Medicine (MCHM) · doi:10.1145/3728424.3760770 · Cited by 1
Decoupled Identity and Attribute Tokenization for Person Re-Identification
Shang, R., Liu, M., Wang, X., Bian, Y., & Wang, Y. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755580 · Cited by 1
Leveraging teleological explanation to support general-purpose AI assessment
Fumagalli, M., & Ferrario, R. (2025) · AI & Society · doi:10.1007/s00146-025-02587-1 · Cited by 2
ESIP: Explicit Surgical Instrument Prompting for Surgical Workflow Recognition
Qiu, Y., Liu, M., He, S.-Y., Zhou, G., Lyu, F., Chen, Y., & Zhou, P. (2025) · IEEE journal of biomedical and health informatics · doi:10.1109/jbhi.2025.3625420
Yang, Y. (2025) · PLoS ONE · doi:10.1371/journal.pone.0337932 · Cited by 4
Alignment Reduces Expressed but Not Encoded Gender Bias: A Unified Framework and Study
Bouchouchi, N., Lesot, M., Laugel, T., Renard, X., Marsala, C., & Detyniecki, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37654-1_21 · Cited by 3
Xie, J., Hu, C., Qi, L., Wang, F., Xu, X., Xiang, H., Zhang, X., Pei, S., Beheshti, A., Dou, W., et al. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792293 · Cited by 3
An innovative hybrid investment model using large language models and genetic algorithms
Huang, C., Yan, Y., Hong, T., & Wu, H.-C. (2026) · Investment Analysts Journal · doi:10.1080/10293523.2026.2644790 · Cited by 1
A probabilistic model of hype reveals a rhetorical shift in biomedicine
Mishra, A., Diesner, J., & Torvik, V. I. (2026) · Quantitative Science Studies · doi:10.1162/qss.a.482 · Cited by 1
Improving LLM-Assisted Domain-Specific Design Tasks Through Domain-Structured Persona Prompting
Zhao, S., & Ji, D. (2026) · Applied Sciences · doi:10.3390/app16125770 · Cited by 1
Chen, S., Xiong, L., Chow, M., & Tang, Y. (2026) · Cyber-Physical Energy Systems · doi:10.1016/j.cpes.2026.04.003 · Cited by 2
Specializing Pre-trained Language Models for Better Relational Reasoning via Network Pruning
Ren, S., & Zhu, K. Q. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.169 · Cited by 5
Alkhaldi, T., Chu, C., & Kurohashi, S. (2022) · Journal of Natural Language Processing · doi:10.5715/jnlp.29.762 · Cited by 2
Invited Paper: Common Public Knowledge for Enhancing Machine Learning Data Sets
Ilani, A., & Dolev, S. (2023) · Workshop on Advanced Tools, Programming Languages, and PLatforms for Implementing and Evaluating Algorithms for Distributed Systems (ApPLIED) · doi:10.1145/3584684.3597263 · Cited by 1
Word-level Prefix/Suffix Sense Detection: A Case Study on Negation Sense with Few-shot Learning
Li, Y., Li, Z., Chen, Y., & Li, S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.484 · Cited by 1
On Conditional and Compositional Language Model Differentiable Prompting
Pilault, J., Liu, C., Bansal, M., & Dreyer, M. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/460 · Cited by 4
Exploring Prompts in Few-Shot Cross-Linguistic Topic Classification Scenarios
Zhang, Z., Liu, S., & Cheng, J. (2023) · Applied Sciences · doi:10.3390/app13179944 · Cited by 1
Emotional Polar Predictions Based on Matching Prompts
Yang, Z., Xia, Y., Xu, H., & Zhao, P. (2023) · International Conference on Intelligent Computing and Signal Processing (ICSP) · doi:10.1109/icsp58490.2023.10248619 · Cited by 1
Li, J., Wang, C., Chen, Y., Liu, K., & Zhao, J. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44693-1_52 · Cited by 1
ReadPrompt: A Readable Prompting Method for Reliable Knowledge Probing
Wang, Z., Ye, L., Wang, H., Kwan, W.-C., Ho, D., & Wong, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.501 · Cited by 10
Identifying Key Terms in Prompts for Relevance Evaluation with GPT Models
Choi, J. (2024) · International Journal on Natural Language Computing · doi:10.5121/ijnlc.2024.13201 · Cited by 4
Luo, M., Gokhale, T., Varshney, N., Yang, Y., & Baral, C. R. (2024) · Synthesis lectures on computer vision · doi:10.1007/978-3-031-57816-8_5 · Cited by 1
Large Language Models for Binary Health-Related Question Answering: A Zero- and Few-Shot Evaluation
Fernández-Pichel, M., Losada, D. E., & Pichel, J. C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-63772-8_29 · Cited by 3
Aspect-level Sentiment Analysis based on Prompt Templates and External Knowledge
Wang, P., Zhang, L., Zheng, Q., & Chen, H. (2024) · International Conference on Computer Engineering and Application (ICCEA) · doi:10.1109/iccea62105.2024.10604075 · Cited by 1
Leveraging LMS Technology to Deliver Formative Assessment in the Post-Pandemic Era
Cassels, A. M. (2024) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-2885-9.ch011 · Cited by 3
ToFC: Tree-of-Fact with Continued Best-First Search for Commonsense Reasoning
Dai, L., & Qin, K. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10649921 · Cited by 1
Enhanced Prompt Learning for Few-shot Text Classification Method
Zio, E., Rossi, M., Garcia, E., Shuqin, Y., & Guangwei, Z. (2024) · Scientific insights and discoveries review · doi:10.59782/sidr.v4i1.78 · Cited by 1
PreAdapter: Pre-training Language Models on Knowledge Graphs
Omeliyanenko, J., Hotho, A., & Schlör, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-77850-6_12 · Cited by 2
DALSCLIP: Domain aggregation via learning stronger domain-invariant features for CLIP
Zhang, Y., Wang, J., Tang, H., & Qin, R. (2024) · Image and Vision Computing · doi:10.1016/j.imavis.2024.105359 · Cited by 4
A Scene-Aware Prompt Template of Chinese Text Data Augmentation Method Based on Large Language Model
Kong, L., Wang, G., He, J., Wang, Y., & Gao, Y. (2024) · IEEE International Conference on Pattern Recognition and Machine Learning (PRML) · doi:10.1109/prml62565.2024.10779819 · Cited by 1
Applying Variational Quantum Classifier on Acceptability Judgements: A QNLP Experiment
Guarasci, R., Buonaiuto, G., Pietro, G. D., & Esposito, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-81247-7_8 · Cited by 1
Integrating Classical Planners with GPT-Based Planning Policies
Tummolo, M., Rossetti, N., Gerevini, A., Olivato, M., Putelli, L., & Serina, I. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-80607-0_24 · Cited by 4
Enhanced image description generation based on key knowledge prompt
Mo, J., He, J., & Yuan, H. (2025) · Signal Image and Video Processing · doi:10.1007/s11760-025-04079-7 · Cited by 1
Correlation-Guided Image-to-Video Transfer Learning for Video Recognition
Hu, W. D., Gao, X., Niu, S., Zhao, H. Y., & Feng, G. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-6596-9_22 · Cited by 2
Prompt Engineering for Safety Red-Teaming: A Case Study on Text-to-Image Diffusion Models
Chen, P., & Liu, S. (2025) · Springer eBooks · doi:10.1007/978-3-031-76770-8_18 · Cited by 1
Prompt-Based Relation Extraction By Reasoning with Contextual Knowledge
Zhang, H., Zhang, X., Cai, M., Wang, W., & Song, Y. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-8186-0_5 · Cited by 1
Multimodales Fenster in die Vergangenheit der ehemaligen Vauban-Festung Saarlouis mittels ChatGPT
Chase, A. V., Chikobava, M., Deru, M., Hauck, C., Poller, P., Ndiaye, A., Baus, J., Schmeyer, T. A., Gampfer, R., & Brandherm, B. (2025) · Angewandte Wirtschaftsinformatik · doi:10.1007/978-3-658-47311-2_9 · Cited by 1
Zhao, Y., Hao, X., Wang, Z., Li, X., Liang, J., & Li, R. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-4158-4_14 · Cited by 1
Metric Sentiment Learning for Label Representation
Song, C., Cai, F., Zheng, J., Chen, W., & Pan, Z. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482369 · Cited by 1
Unsupervised Ranking and Aggregation of Label Descriptions for Zero-Shot Classifiers
Basile, A., Franco-Salvador, M., & Rosso, P. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-08473-7_11 · Cited by 3
Aggarwal, C. C. (2022) · Springer eBooks · doi:10.1007/978-3-030-96623-2_14 · Cited by 1
DPNPED: Dynamic Perception Network for Polysemous Event Trigger Detection
Xu, J., & Sun, M. (2022) · IEEE Access · doi:10.1109/access.2022.3210697 · Cited by 1
Large Language Models and Their Current Use Cases
Özçift, A. (2023) · doi:10.59287/pcse.346 · Cited by 1
Two Examples are Better than One: Context Regularization for Gradient-based Prompt Tuning
Ha, H., Jung, S., Park, J., Seo, M., Hwang, S.-W., & Chun, B.-G. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.206 · Cited by 1
Is It Smaller Than a Tennis Ball? Language Models Play the Game of Twenty Questions
Bruyn, M. D., Lotfi, E., Buhmann, J., & Daelemans, W. M. P. (2022) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2022.blackboxnlp-1.7 · Cited by 4
Knowledge Representation Learning and Knowledge-Guided NLP
Han, X., Chen, W., Liu, Z., Lin, Y., & Sun, M. (2023) · Springer eBooks · doi:10.1007/978-981-99-1600-9_9 · Cited by 1
Unnatural language processing: How do language models handle machine-generated prompts?
Kervadec, C., Franzon, F., & Baroni, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.959 · Cited by 8
Xia, X., Liu, Y., & Fu, L. (2024) · Computer Speech & Language · doi:10.1016/j.csl.2024.101650 · Cited by 1
Knowledge enhanced bottom-up affordance grounding for robotic interaction
Wen, Q., Li, X., & Xiao, J. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.2097 · Cited by 4
Exploring prompts to elicit memorization in masked language model-based named entity recognition
Xia, Y., Sedova, A., de Araujo, P. H. L., Kougia, V., Nußbaumer, L., & Roth, B. (2025) · PLoS ONE · doi:10.1371/journal.pone.0330877 · Cited by 1
Probing Pre-Trained Language Models for Disease KnowledgePreprint
Alghanmi, I., Espinosa-Anke, L., & Schockaert, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.07285 · Cited by 4
Towards Language Modelling in the Speech Domain Using Sub-word Linguistic UnitsPreprint
Katakkar, A., & Black, A. W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.00610 · Cited by 1
Toward context-based text-to-3D scene generation
Henlein, A. (2022) · Publication Server of Goethe University Frankfurt am Main (Goethe University Frankfurt) · doi:10.21248/gups.73448 · Cited by 1
Studies in conversational AI : multilingual capabilities, world knowledge, and evaluation strategies
Bruyn, M. D. (2024) · doi:10.63028/10067/2023030151162165141 · Cited by 1
scKEPLM: Knowledge enhanced large-scale pre-trained language model for single-cell transcriptomicsPreprint
Li, Y., Qiao, G., & Wang, G. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.07.09.602633 · Cited by 2
Metadata
- Authors
- Jiang, Z., Xu, F. F., Araki, J., & Neubig, G.
- Affiliations
- Carnegie Mellon University (US)
- Robert Bosch (United States) (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Pages
- 423-438
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2991382858
- Cited by
- 1,002
- References
- 52
- Citations by year
- 2026: 76 · 2025: 169 · 2024: 224 · 2023: 285 · 2022: 144 · 2021: 78 · 2020: 25
- FWCI
- 52.35
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1815287)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- prompt generation, ensemble methods
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.