How Can We Know What Language Models Know?
Open accessThe DOI matches its Crossref record; 51 of the 52 references it cites were found in a registry or an index.
Abstract
Recent work has presented intriguing results examining the knowledge contained in language models (LMs) by having the LM fill in the blanks of prompts such as “ Obama is a __ by profession”. These prompts are usually manually created, and quite possibly sub-optimal; another prompt such as “ Obama worked as a __ ” may result in more accurately predicting the correct profession. Because of this, given an inappropriate prompt, we might fail to retrieve facts that the LM does know, and thus any given prompt only provides a lower bound estimate of the knowledge contained in an LM. In this paper, we attempt to more accurately estimate the knowledge contained in LMs by automatically discovering better prompts to use in this querying process. Specifically, we propose mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different prompts. Extensive experiments on the LAMA benchmark for extracting relational knowledge from LMs demonstrate that our methods can improve accuracy from 31.1% to 39.6%, providing a tighter lower bound on what LMs know. We have released the code and the resulting LM Prompt And Query Archive (LPAQA) at https://github.com/jzbjyb/LPAQA .
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The ability of Large Language Models to retrieve factual knowledge as explicit knowledge bases [53-56, 56-61] has been stirred discussed, which presents an opportunity to construct more comprehensive knowledge graphs within recommender systems.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [56]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“even experienced prompt designers may fail to manually discover optimal prompts [52].”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [52]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Beyond this, we found models trained with Toolformer to often be sensitive to the exact wording of their input when deciding whether or not to call an API; this is perhaps unsurprising given that LMs are known to be very sensitive to the prompt they are provided with in both zeroand few-shot settings (Jiang et al., 2020;Schick and Schütze, 2021a).”
From Toolformer: Language Models Can Teach Themselves to Use Tools (2023) · cites this paper as Jiang et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“There is some evidence to suggest that formats that occur commonly in the training data will lead to better performance (Jiang et al., 2020).”
From The Prompt Report: A Systematic Survey of Prompt Engineering Techniques (2024) · cites this paper as Jiang et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Jiang et al. [52]'s Mine approach is a mining-based method to automatically find templates given a set of training inputs x and outputs y.”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [52]
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“Prompts where the hidden vectors learned through a language modeling objective are then used in downstream language understanding systems (Dai and Le, 2015; Melamud et al., 2016; Peters et al., 2018; Devlin et al., 2019).”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
A Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1014 · Cited by 1,926
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834
ERNIE: Enhanced Language Representation with Informative Entities
Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., & Liu, Q. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1139 · Cited by 1,482 · cited 5×
“Similar extensions have been applied to pre-trained LMs like BERT, where contextualized representations are enhancedwith entity embeddings (Zhang et al., 2019; Peters et al., 2019; Pörner et al., 2019).”
MethodInfluentialOpen information extraction from the web
Banko, M., Cafarella, M., Soderland, S. G., Broadhead, M., & Etzioni, O. (2007) · International Joint Conference on Artificial Intelligence · Cited by 1,398
BERT Rediscovers the Classical NLP Pipeline
Tenney, I., Das, D., & Pavlick, E. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1452 · Cited by 1,307
(2019) · doi:10.18653/v1/p19-1356 · Cited by 1,291 · cited 1×
“…a broad suite of NLP tasks are used to analyze both syntactic and semantic properties, providing evidence that contextualized representations encode linguistic knowledge in different layers (Hewitt and Manning, 2019; Tenney et al., 2019a; Tenney et al., 2019b; Jawahar et al., 2019; Goldberg, 2019).”
BackgroundSnowball: extracting relations from large plain-text collections
Agichtein, E., & Gravano, L. (2000) · Proceedings of the fifth ACM conference on Digital libraries · doi:10.1145/336597.336644 · Cited by 1,147
Research works printed but not linked: 7
- Andrew M. Dai and Quoc V. Le. 2015. Semisupervised sequence learning. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada , pages 3079-3087.
- Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A diversitypromoting objective function for neural conversation models. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016 , pages 110-119.
- Gábor Melis, Chris Dyer, and Phil Blunsom. 2018. On the state of the art of evaluation in neural language models. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings .
- Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018. Regularizing and optimizing LSTM language models. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings .
- 1-6, 2018, Volume 1 (Long Papers) , pages 22272237.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog , 1(8).
- Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019b. What do you learn from context? Probing for sentence structure in contextualized word representations. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 .
Cited by
Items 701–800 of 1,040
Retentive or forgetful? diving into the knowledge memorizing mechanism of language models
Cao, B., Tang, Q., Lin, H., Jiang, S., Dong, B., Han, X., Chen, J., Wang, T., & Sun, L. (2024) · Proceedings of the Language Resources and Evaluation Conference · doi:10.63317/4h8nkuscdqqa · Cited by 4
Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data
Shum, K., Diao, S., & Zhang, T. (2023) · arXiv · doi:10.48550/arxiv.2302.12822 · Cited by 3
“Existing prompting methods can be categorized into two types based on their nature: 1) discrete prompts (Wallace et al., 2019; Shin et al., 2020; Jiang et al., 2020; Yuan et al., 2021; Haviv et al., 2021; Gao et al., 2021; Ben-David et al., 2022; Davison et al., 2019; Su et al., 2022; Diao et al.,…”
BackgroundTang, Y., Situ, J., & Huang, Y. (2024) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/3675094.3678995 · Cited by 7
PromptHC: Multi-attention prompt guided haze-weather crowd counting
Shen, J., Tong, L., Li, S., & Kong, W. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128023 · Cited by 3
AFMIP-Net: Adaptive feature modulation and implicit prompt network for retinal vessel segmentation
Xie, Y., Shang, J., Deng, R., Cao, B., Li, S., & Tang, X. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.131106 · Cited by 3
Visual Consensus Prompting for Co-Salient Object Detection
Wang, J., Yu, N., Zhang, Z., & Han, Y. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00896 · Cited by 3
ReqGPT: a fine-tuned large language model for generating requirements documents
Schiller, K. A., Haddad, M.-S., & Seibel, A. (2025) · Proceedings of the Design Society · doi:10.1017/pds.2025.10288 · Cited by 8
Privacy-preserving personalized federated prompt learning for vision-language models
Yinan, W., Ren, Y., Guo, Z., & Huang, M. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.108220 · Cited by 5
Self-attention vector output similarities reveal how machines pay attention
Halevi, T., Tzach, Y., Gross, R. D., Rosner, S., & Kanter, I. (2026) · Physica A Statistical Mechanics and its Applications · doi:10.1016/j.physa.2026.131363 · Cited by 3
A large-scale evaluation of commonsense knowledge in humans and large language models
Nguyen, T. D., Watts, D. J., & Whiting, M. E. (2026) · PNAS Nexus · doi:10.1093/pnasnexus/pgag029 · Cited by 10
Lan, G., Hu, M., Su, Y., Sun, Y., & Zhang, Y. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.115892 · Cited by 3
Bridging Perception, Memory, and Inference through Semantic Relations
Björklund, J., Lindström, A. D., & Drewes, F. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.719 · Cited by 3
Decoding Symbolism in Language Models
Guo, M., Hwa, R., & Kovashka, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.186 · Cited by 7
Li, D., & Xu, F. (2024) · IEEE International Conference on Sensors, Electronics and Computer Engineering (ICSECE) · doi:10.1109/icsece61636.2024.10729340 · Cited by 5
Self-citationFine-Tuning Pre-trained Model with Optimizable Prompt Learning for Code Vulnerability Detection
Chang, W., Ye, C., & Zhou, H. (2024) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre62328.2024.00021 · Cited by 6
On Calibration of Prompt Learning Using Temperature Scaling
Nguyen, K.-B., & Park, C. J. (2025) · IEEE Access · doi:10.1109/access.2025.3538617 · Cited by 4
RaT2IGen: Relation-aware Text-to-image Generation via Learnable Prompt
Ye, Z., He, X., & Peng, Y. (2025) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3726527 · Cited by 5
Segmenting Anything in the Dark via Depth Perception
Liu, P., Deng, J., Duan, L., Li, W., & Lv, F. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3557612 · Cited by 5
RelBERT: Embedding relations with language models
Ushio, A., Camacho-Collados, J., & Schockaert, S. (2025) · Artificial Intelligence · doi:10.1016/j.artint.2025.104359 · Cited by 5
Ahmad, K. M., Liu, Q., Khan, A. A., Pang, R., Dai, T., & Gan, Y. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107711 · Cited by 4
DDPT: Diffusion-Driven Prompt Tuning for Large Language Model Code Generation
Li, J., Hyun, S., & Babar, M. A. (2025) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/cain66642.2025.00030 · Cited by 5
Joint Scheduling of Causal Prompts and Tasks for Multi-Task Learning
Li, C., Qin, J., Cui, J., Liu, Z., Hu, N., & Liao, Q. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02339 · Cited by 5
Gili, K., Heuton, K., Shah, A., Hammer, D., & Hughes, M. C. (2025) · Physical Review Physics Education Research · doi:10.1103/v6gw-lv4c · Cited by 3
Unsupervised Dual Modality Prompt Learning for Facial Expression Recognition
Shahid, M. (2023) · International Conference on Computer and Communication Systems (ICCCS) · doi:10.1109/icccs57501.2023.10151313 · Cited by 3
When Gradient Descent Meets Derivative-Free Optimization: A Match Made in Black-Box Scenario
Han, C., Cui, L., Zhu, R., Wang, J., Chen, N., Sun, Q., Li, X., & Gao, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.55 · Cited by 7
Knowledge Base Completion for Long-Tail Entities
Chen, L., Razniewski, S., & Weikum, G. (2023) · doi:10.18653/v1/2023.matching-1.8 · Cited by 13
Zhen, Z., Wang, L., Gu, D., Wu, H. K., Janfada, B., & Bidgoli, B. M. (2023) · International Journal of Information Technologies and Systems Approach · doi:10.4018/ijitsa.328681 · Cited by 3
Unsupervised Improvement of Factual Knowledge in Language Models
Sadeq, N., Kang, B., Lamba, P., & McAuley, J. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.215 · Cited by 6
Sai, C., Sadiq, S. W., Han, L., Demartini, G., & Rinderle‐Ma, S. (2024) · Lecture notes in business information processing · doi:10.1007/978-3-031-59465-6_11 · Cited by 3
Israni, A. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3638193 · Cited by 3
Enhancing GPT-Based Planning Policies by Model-Based Plan Validation
Rossetti, N., Tummolo, M., Gerevini, A., Olivato, M., Putelli, L., & Serina, I. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-71170-1_26 · Cited by 9
APL: Anchor-Based Prompt Learning for One-Stage Weakly Supervised Referring Expression Comprehension
Luo, Y., Ji, J., Chen, X., Zhang, Y., Ren, T., & Luo, G. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72624-8_12 · Cited by 3
Wang, B., Wang, Z., Xiang, W., & Mo, Y. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3527135 · Cited by 4
Gao, J., Hao, W. U., Cheung, Y., Cao, J., Yu, H., & Zhang, Y. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714687 · Cited by 7
Unfreeze with Care: Space-Efficient Fine-Tuning of Semantic Parsing Models
Sun, W., Khan, H., Mesnards, N. G. D., Rubino, M., & Arkoudas, K. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3511942 · Cited by 5
Chandak, S., Zhang, L., Brown, C., & Huang, L. (2022) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2022.bionlp-1.40 · Cited by 3
ASCM: An Answer Space Clustered Prompting Method without Answer Engineering
Wang, Z., Yang, Y., Xi, Z., Ma, B., Wang, L., Dong, R., & Anwar, A. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.193 · Cited by 4
Using Bidirectional Prompt Learning in NLP Few Shot Tasks
Li, D., & Ye, S. (2023) · Frontiers in Computing and Intelligent Systems · doi:10.54097/fcis.v3i1.6362 · Cited by 3
VIPHY: Probing “Visible” Physical Commonsense Knowledge
Singh, S., Qasemi, E., & Chen, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.473 · Cited by 9
Selitskiy, S., Inoue, C., Schetinin, V., & Jakaite, L. (2023) · International Conference on Social Networks Analysis, Management and Security (SNAMS) · doi:10.1109/snams60348.2023.10375471 · Cited by 5
Automatic Robotic Development through Collaborative Framework by Large Language Models
Luan, Z., Lai, Y., Huang, R., Lan, X., Chen, L., Chen, B., Yan, Y., & Shang, Y. (2023) · China Automation Congress (CAC) · doi:10.1109/cac59555.2023.10450432 · Cited by 5
Self-refined variational transformer for image-conditioned layout generation
Cao, Y., Liu, C., Ma, Y., Zhou, M., Ge, T., Jiang, Y., & Xie, H. (2024) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02355-5 · Cited by 3
Learning Event Extraction From a Few Guideline Examples
Hong, R., Zhang, H., Yu, X., & Zhang, C. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3202123 · Cited by 5
Hosseini, M., & Caragea, C. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26514 · Cited by 3
WikiOmnia: filtration and evaluation of the generated QA corpus on the whole Russian Wikipedia
Pisarevskaya, D., & Shavrina, T. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.10 · Cited by 5
Auto-verbalizer filtering for prompt-based aspect category detection
Xian, Y., Qin, Y., & Xiang, Y. (2024) · Frontiers in Physics · doi:10.3389/fphy.2024.1361695 · Cited by 3
Analyzing COVID-19 Tweets with Transformer-based Language ModelsPreprint
Feldman, P., Tiwari, S., Cheah, C. S. L., Foulds, J. R., & Pan, S. (2021) · arXiv (Cornell University) · doi:10.13016/m2j3r1-ktks · Cited by 12
GraphPrompt: Biomedical Entity Normalization Using Graph-based Prompt TemplatesPreprint
Zhang, J., Wang, Z., Zhang, S., Bhalerao, M. M., Liu, Y., Zhu, D., & Wang, S. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.11.29.470486 · Cited by 3
Beyond Extraction: Contextualising Tabular Data for Efficient Summarisation by Language ModelsPreprint
Allu, U., Ahmed, B., & Tripathi, V. (2024) · doi:10.36227/techrxiv.170792474.42605726/v1 · Cited by 4
Efficient Prompt Optimization Through the Lens of Best Arm Identification
Shi, C., Yang, K., Chen, Z., Li, J., Yang, J., & Shen, C. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-3161 · Cited by 3
“Instead, many proposed methods rely on the pipeline of first generating a pool of candidate prompts and then selecting from it [35, 90, 79, 59, 28].”
Instruction-tuned Language Models are Better Knowledge Learners
Jiang, Z., Sun, Z., Shi, W., Rodriguez, P., Zhou, C., Neubig, G., Lin, X. V., Yih, W.-T., & Iyer, S. (2024) · arXiv · doi:10.48550/arxiv.2402.12847 · Cited by 3
Self-citationMitigating Word Bias in Zero-shot Prompt-based Classifiers
Liusie, A., Manakul, P., & Gales, M. (2023) · Findings of the Association for Computational Linguistics: IJCNLP-AACL 2023 (Findings) · doi:10.18653/v1/2023.findings-ijcnlp.29 · Cited by 2
Pu, J., Cheng, L., Fan, L., Lv, T., & Zhang, R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.590 · Cited by 3
Topic-Oriented Controlled Text Generation for Social Networks
Yang, Z., Jiang, H., Deng, A., & Li, Y. (2024) · Journal of Signal Processing Systems · doi:10.1007/s11265-023-01907-2 · Cited by 4
P-Distill: Efficient and Effective Prompt Tuning Using Knowledge Distillation
Won, H.-S., Choi, J.-Y., Zaman, N., Aliyeva, D., & Kim, K.-M. (2025) · Applied Sciences · doi:10.3390/app15052420 · Cited by 2
Dertli, Z. G., & Yıldız, B. (2025) · Anatolian Journal of Education · doi:10.29333/aje.2025.1015a · Cited by 3
Cross-Modality Prompts: Few-Shot Multi-Label Recognition With Single-Label Training
Ding, Z., Zhou, Z., Chen, H., Hao, T., Xiong, Y., Zhao, S., Zhang, Q., & Han, J. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3557700 · Cited by 3
Entangle-then-disentangle: a novel approach for enhancing large vision-language model
Yuan, J., Zheng, H., Yu, H., & Luo, X. (2025) · Machine Learning · doi:10.1007/s10994-025-06811-3 · Cited by 2
Automated Essential Concept Discovery for Few-Shot Out-of-Distribution Detection
Chen, G., Horstmann, K. A., Wang, Z., & You, F. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00381 · Cited by 6
An exploration into the nature of ChatGPT’s mathematical knowledge
Marco, N., & Stylianides, A. J. (2025) · International Journal of Mathematical Education in Science and Technology · doi:10.1080/0020739x.2025.2543817 · Cited by 5
Particle Swarm Optimization Meets Large Language Models
Yu, X. (2025) · International Conference on Computer Communication and Artificial Intelligence (CCAI) · doi:10.1109/ccai65422.2025.11189827 · Cited by 3
Explainable Multi-Hop Question Answering: A Rationale-Based Approach
Han, K., Jang, Y., & Kim, H. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9110273 · Cited by 3
FedQuad: Adaptive Layer-Wise LoRA Deployment and Activation Quantization for Federated Fine-Tuning
Liu, J., Li, R., Xu, H., Ma, Q., Yan, J., & Huang, L. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3637064 · Cited by 11
Diversity covariance-aware prompt learning for vision-language models
Zhou, Z., Dong, S., Ding, C., Gao, X., He, Y., & Gong, Y. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.112806 · Cited by 5
Goethals, S., Rhue, L. A., & Sundararajan, A. (2025) · AI and Ethics · doi:10.1007/s43681-025-00876-5 · Cited by 4
Dual-prompt-based binary matching for open set domain adaptation
Yang, J., JIANG, S.-X., Yao, H., Xu, L., Jin, S., Zhang, H., Xu, Z., & Jiang, J. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.132425 · Cited by 2
Knowledge enhanced prompting for few-shot named entity recognition
Liu, Z., Qin, T., Liu, B., Sun, Q., & Li, S. (2026) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-025-02902-8 · Cited by 2
Select prompting with chain-of-thought paired with large language models
Che, X., Wu, W., Chen, Y., Jiang, L., & Li, Q. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.131511 · Cited by 5
Generalist multimodal AI: A review of architectures, challenges and opportunities
Munikoti, S., Stewart, I., Horawalavithana, S., Kvinge, H., Emerson, T., Thompson, S., & Pazdernik, K. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.132933 · Cited by 9
Single-nodal spontaneous symmetry breaking in NLP models
Rosner, S., Gross, R. D., Koresh, E., & Kanter, I. (2026) · Physica A Statistical Mechanics and its Applications · doi:10.1016/j.physa.2026.131426 · Cited by 4
Braune, J., Conradty, C., Bogner, F. X., & Paul, J. (2026) · Frontiers in Education · doi:10.3389/feduc.2026.1770878 · Cited by 2
More is different in AI—More is the same in physics
Kanter, I. (2026) · Physica A Statistical Mechanics and its Applications · doi:10.1016/j.physa.2026.131534 · Cited by 2
Multi-Classification of Electric Power Metadata based on Prompt-tuning
Liang, X., Zhang, W., Lei, S., Zhang, Y., Xu, M., Peng, L., & Feng, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-23504-7_8 · Cited by 2
Eliciting Affective Events from Language Models by Multiple View Co-prompting
Zhuang, Y., & Riloff, E. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.199 · Cited by 2
ChatGPT backend: A comprehensive analysis
Belgacem, A., Bradai, A., & Beghdad-Bey, K. (2023) · International Symposium on Networks, Computers and Communications (ISNCC) · doi:10.1109/isncc58260.2023.10323792 · Cited by 3
Towards Automating Database Designing
Helskyaho, H. (2023) · Conference of Open Innovations Association (FRUCT) · doi:10.23919/fruct60429.2023.10328153 · Cited by 5
Who Are All The Stochastic Parrots Imitating? They Should Tell Us!
Shaier, S., Hunter, L., & Kann, K. (2023) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2023.ijcnlp-short.13 · Cited by 13
Snapshot Prompt Ensemble for Parameter-Efficient Soft Prompt Transfer
Wu, X., Chen, C. L. P., Li, S., & Zhang, T. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448070 · Cited by 2
A Comprehensive Review of Relation Prediction Techniques in Knowledge Graph
Lu, Y., Yang, S., & Tang, B. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-2991-3_2 · Cited by 2
Cloze Quality Estimation for Language Assessment
Zhang, Z., Mita, M., & Komachi, M. (2024) · Journal of Natural Language Processing · doi:10.5715/jnlp.31.328 · Cited by 2
Intelli-Change Remote Sensing - A Novel Transformer Approach
Vyshnav, D., Gutha, L., Manindra, A. P. V., & Karthikeyan, B. (2024) · International Conference on Data Science and Information System (ICDSIS) · doi:10.1109/icdsis61070.2024.10594026 · Cited by 3
Liu, S., Ao, Z., Chen, P., & Kolmanič, S. (2024) · IEEE Access · doi:10.1109/access.2024.3434720 · Cited by 3
PTE: Prompt tuning with ensemble verbalizers
Liang, L., Wang, G., Lin, C., & Feng, Z. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125600 · Cited by 4
GEM-RAG: Graphical Eigen Memories for Retrieval Augmented Generation
Rappazzo, B., Wang, Y., Ferber, A., & Gomes, C. P. (2024) · International Conference on Machine Learning and Applications · doi:10.1109/icmla61862.2024.00196 · Cited by 5
Agent-Based Control Prompt Tuning for Video-Text Retrieval
Lai, H., Wei, X., Sun, R., & Zhang, T. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3574726 · Cited by 3
Designing Templates for Eliciting Commonsense Knowledge from Pretrained Sequence-to-Sequence Models
Yang, J.-H., Lin, S.-C., Nogueira, R. F., Tsai, M.-F., Wang, C., & Lin, J. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.307 · Cited by 2
Continuous Entailment Patterns for Lexical Inference in Context
Schmitt, M., & Schütze, H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.556 · Cited by 2
On Masked Language Models for Contextual Link Prediction
Brayne, A., Wiatrak, M., & Corneil, D. (2022) · Workshop on Knowledge Extraction and Integration for Deep Learning Architectures: Deep Learning Inside Out (DeeLIO) · doi:10.18653/v1/2022.deelio-1.9 · Cited by 4
Continuous Prompt Enhanced Biomedical Entity Normalization
Lai, Z., Fu, B., Wei, S., & Shi, X. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-17189-5_5 · Cited by 2
Towards Alleviating the Object Bias in Prompt Tuning-based Factual Knowledge Extraction
Wang, Y., Lu, D., Kong, C., & Sang, J. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.270 · Cited by 11
MetaFill: Text Infilling for Meta-Path Generation on Heterogeneous Information Networks
Liu, Z., Duan, K., Yang, J., Xu, H., Zhang, M., & Wang, S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.341 · Cited by 2
Yan, J., Xie, Y., Guo, Y., Wei, Y., Zhang, X.-P. S., & Luan, X. (2023) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-023-00286-5 · Cited by 4
Instance-Aware and Semantic-Guided Prompt for Few-Shot Learning in Large Language Models
Weng, J., Li, D., Deng, Y., Zhang, J., Hu, Y., & Huang, H. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-8148-9_5 · Cited by 2
Evaluating large language models’ ability to generate interpretive arguments
Marji, Z., & Licato, J. (2024) · Argument & Computation · doi:10.3233/aac-230014 · Cited by 2
Web Personalization with Large Language Models: Challenges and Future Trends
Bansal, N., Bala, M., & Sharma, K. (2025) · Algorithms for intelligent systems · doi:10.1007/978-981-97-8669-5_21 · Cited by 3
A Retrieval-Augmented Generation method based on Intent Analysis and Ranking Fusion
Kong, L., Wang, G., Liu, N. K. J., Bai, T., & He, J. (2024) · International Conference on Data Science and Information Technology (DSIT) · doi:10.1109/dsit61374.2024.10881002 · Cited by 3
WhisPrompt: Audio classification of Chinese opera genres by transferring time-series features
Shi, B., Wang, H., Qiu, J., & Li, X. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127301 · Cited by 5
Hsieh, P., & Lee, W. (2025) · Expert Systems · doi:10.1111/exsy.70049 · Cited by 3
Verbalization diversification for relation extraction with long-tailed entity augmentation
Liang, Y., Jiang, T., Xin, Y., Yan, C., Wang, L., Wang, X., & He, L. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130771 · Cited by 2
Metadata
- Authors
- Jiang, Z., Xu, F. F., Araki, J., & Neubig, G.
- Affiliations
- Carnegie Mellon University (US)
- Robert Bosch (United States) (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Pages
- 423-438
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2991382858
- Cited by
- 1,002
- References
- 52
- Citations by year
- 2026: 76 · 2025: 169 · 2024: 224 · 2023: 285 · 2022: 144 · 2021: 78 · 2020: 25
- FWCI
- 52.35
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1815287)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- prompt generation, ensemble methods
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.