How Can We Know What Language Models Know?
Open accessThe DOI matches its Crossref record; 51 of the 52 references it cites were found in a registry or an index.
Abstract
Recent work has presented intriguing results examining the knowledge contained in language models (LMs) by having the LM fill in the blanks of prompts such as “ Obama is a __ by profession”. These prompts are usually manually created, and quite possibly sub-optimal; another prompt such as “ Obama worked as a __ ” may result in more accurately predicting the correct profession. Because of this, given an inappropriate prompt, we might fail to retrieve facts that the LM does know, and thus any given prompt only provides a lower bound estimate of the knowledge contained in an LM. In this paper, we attempt to more accurately estimate the knowledge contained in LMs by automatically discovering better prompts to use in this querying process. Specifically, we propose mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different prompts. Extensive experiments on the LAMA benchmark for extracting relational knowledge from LMs demonstrate that our methods can improve accuracy from 31.1% to 39.6%, providing a tighter lower bound on what LMs know. We have released the code and the resulting LM Prompt And Query Archive (LPAQA) at https://github.com/jzbjyb/LPAQA .
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The ability of Large Language Models to retrieve factual knowledge as explicit knowledge bases [53-56, 56-61] has been stirred discussed, which presents an opportunity to construct more comprehensive knowledge graphs within recommender systems.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [56]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“even experienced prompt designers may fail to manually discover optimal prompts [52].”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [52]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Beyond this, we found models trained with Toolformer to often be sensitive to the exact wording of their input when deciding whether or not to call an API; this is perhaps unsurprising given that LMs are known to be very sensitive to the prompt they are provided with in both zeroand few-shot settings (Jiang et al., 2020;Schick and Schütze, 2021a).”
From Toolformer: Language Models Can Teach Themselves to Use Tools (2023) · cites this paper as Jiang et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“There is some evidence to suggest that formats that occur commonly in the training data will lead to better performance (Jiang et al., 2020).”
From The Prompt Report: A Systematic Survey of Prompt Engineering Techniques (2024) · cites this paper as Jiang et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Jiang et al. [52]'s Mine approach is a mining-based method to automatically find templates given a set of training inputs x and outputs y.”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [52]
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“Prompts where the hidden vectors learned through a language modeling objective are then used in downstream language understanding systems (Dai and Le, 2015; Melamud et al., 2016; Peters et al., 2018; Devlin et al., 2019).”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
A Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1014 · Cited by 1,926
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834
ERNIE: Enhanced Language Representation with Informative Entities
Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., & Liu, Q. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1139 · Cited by 1,482 · cited 5×
“Similar extensions have been applied to pre-trained LMs like BERT, where contextualized representations are enhancedwith entity embeddings (Zhang et al., 2019; Peters et al., 2019; Pörner et al., 2019).”
MethodInfluentialOpen information extraction from the web
Banko, M., Cafarella, M., Soderland, S. G., Broadhead, M., & Etzioni, O. (2007) · International Joint Conference on Artificial Intelligence · Cited by 1,398
BERT Rediscovers the Classical NLP Pipeline
Tenney, I., Das, D., & Pavlick, E. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1452 · Cited by 1,307
(2019) · doi:10.18653/v1/p19-1356 · Cited by 1,291 · cited 1×
“…a broad suite of NLP tasks are used to analyze both syntactic and semantic properties, providing evidence that contextualized representations encode linguistic knowledge in different layers (Hewitt and Manning, 2019; Tenney et al., 2019a; Tenney et al., 2019b; Jawahar et al., 2019; Goldberg, 2019).”
BackgroundSnowball: extracting relations from large plain-text collections
Agichtein, E., & Gravano, L. (2000) · Proceedings of the fifth ACM conference on Digital libraries · doi:10.1145/336597.336644 · Cited by 1,147
Research works printed but not linked: 7
- Andrew M. Dai and Quoc V. Le. 2015. Semisupervised sequence learning. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada , pages 3079-3087.
- Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A diversitypromoting objective function for neural conversation models. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016 , pages 110-119.
- Gábor Melis, Chris Dyer, and Phil Blunsom. 2018. On the state of the art of evaluation in neural language models. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings .
- Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018. Regularizing and optimizing LSTM language models. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings .
- 1-6, 2018, Volume 1 (Long Papers) , pages 22272237.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog , 1(8).
- Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019b. What do you learn from context? Probing for sentence structure in contextualized word representations. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 .
Cited by
Items 601–700 of 1,040
No clues good clues: out of context Lexical Relation Classification
Pitarch, L., Bernad, J., Dranca, L., Bobed, C., & Gracia, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.308 · Cited by 6
Exploring Lottery Prompts for Pre-trained Language Models
Chen, Y., Ding, N., Wang, X., Hu, S., Zheng, H.-T., Liu, Z., & Xie, P. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.860 · Cited by 11
Tree Prompting: Efficient Task Adaptation without Fine-Tuning
Singh, C., Morris, J. X., Rush, A. M., Gao, J., & Deng, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.384 · Cited by 33
A Chinese Few-Shot Text Classification Method Utilizing Improved Prompt Learning and Unlabeled Data
Hu, T., Chen, Z., Ge, J., Yang, Z., & Ji-chao, X. (2023) · Applied Sciences · doi:10.3390/app13053334 · Cited by 7
Long-Tailed Question Answering in an Open World
Dai, Y. Q., Lang, H., Zheng, Y., Huang, F., & Li, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.351 · Cited by 12
Plug-and-Play Document Modules for Pre-trained Models
Xiao, C., Zhang, Z., Han, X., Chan, C.-M., Lin, Y., Liu, Z., Li, X., Li, Z., Cao, Z., & Sun, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.875 · Cited by 18
Inducer-tuning: Connecting Prefix-tuning and Adapter-tuning
Chen, Y., Hazarika, D., Namazifar, M., Liu, Y., Jin, D., & Hakkani-Tür, D. Z. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.50 · Cited by 12
Few-shot Query-Focused Summarization with Prefix-Merging
Yuan, R., Wang, Z., Cao, Z., & Li, W. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.243 · Cited by 11
Supporting vision-language model few-shot inference with confounder-pruned knowledge prompt
Li, J., Mo, W., Song, F., Sun, C., Qiang, W., Su, B., & Zheng, C. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107173 · Cited by 15
Rethinking visual prompt learning as masked visual token modeling
Liao, N., Shi, B., Zhang, X., Cao, M., Yan, J., & Tian, Q. C. (2025) · Artificial Intelligence · doi:10.1016/j.artint.2025.104417 · Cited by 9
Multilingual Fact LinkingPreprint
Kolluru, K., Rezk, M., Verga, P., Cohen, W. W., & Talukdar, P. P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.14364 · Cited by 8
Tian, Z., Zhang, H., & Wang, Y. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112646 · Cited by 5
FGVIrony: A Chinese Dataset of Fine-grained Verbal Irony
Wen, Z., Wang, R., Wang, Q., Gui, L., Long, Y., Chen, S., Liang, B., Yang, M., & Xu, R. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104169 · Cited by 5
Hang, T., Liu, S., Feng, J., Djigal, H., & Huang, J. (2025) · ACM Computing Surveys · doi:10.1145/3746281 · Cited by 11
Discrete prompt optimization using genetic algorithm for secure Python code generation
Tony, C., Pintor, M., Kretschmann, M., & Scandariato, R. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112682 · Cited by 7
Advancing Healthcare with Large Language Models: Techniques and Application
Hu, Z., Peng, Z., Bi, Z., Shen, Q., Liu, Z., Lou, J., & Luo, X. (2025) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2025.125540 · Cited by 12
Geographical Erasure in Language Generation
Schwöbel, P., Golebiowski, J., Donini, M., Archambeau, C., & Pruthi, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.823 · Cited by 24
A novel prompt-tuning method: Incorporating scenario-specific concepts into a verbalizer
Ma, Y., Luo, S., Shang, Y.-M., Li, Z., & Liu, Y. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123204 · Cited by 6
Jo, E., Yoo, H., Kim, J.-H., Kim, Y.-M., Song, S., & Joo, H. J. (2024) · JMIR Formative Research · doi:10.2196/47814 · Cited by 7
SCP: Soft Conditional Prompt Learning for Aerial Video Action Recognition
Wang, X., Xian, R., Guan, T., Liu, F., & Manocha, D. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802074 · Cited by 5
Rehearsal-free continual few-shot relation extraction via contrastive weighted prompts
Yang, F., Ren, M., Kong, D., Liu, S., & Fu, Z. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129741 · Cited by 5
Bringing Two Worlds Together: The Convergence of Large Language Models and Knowledge Graphs
Cai, L., Kang, Y., Yu, C., Fu, Y., Zhang, H., & Zhao, Y. (2024) · International Conference on Automation, Robotics and Computer Engineering (ICARCE) · doi:10.1109/icarce63054.2024.00046 · Cited by 9
Multi-task prompt adapter for vision-language model
Wei, Z., Han, J., Ding, C., Wang, C., & Luo, L. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113254 · Cited by 6
Multilingual capabilities of GPT: A study of structural ambiguity
Yoo, M. H., Kim, J.-S., & Song, S. (2025) · PLoS ONE · doi:10.1371/journal.pone.0326943 · Cited by 10
Zero-Shot Slot Filling with Slot-Prefix Prompting and Attention Relationship Descriptor
Luo, Q., & Liu, L. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26566 · Cited by 13
Improved Word Sense Disambiguation via Prompt-based Contextual Word Representation
He, Q., Zhang, J., & Huang, X. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191929 · Cited by 5
Vector-Quantized Input-Contextualized Soft Prompts for Natural Language Understanding
Bhardwaj, R., Saha, A., Hoi, S. C. H., & Poria, S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.455 · Cited by 13
SPE: Symmetrical Prompt Enhancement for Fact Probing
Li, Y., Che, T., Wang, Y., Jiang, Z., Xiong, C., & Chaturvedi, S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.803 · Cited by 8
Self-citationCan Language Models Serve as Temporal Knowledge Bases?
Zhao, R., Zhao, F., Xu, G., Zhang, S., & Jin, H. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.147 · Cited by 6
Soft Prompt Transfer for Zero-Shot and Few-Shot Learning in EHR Understanding
Wang, Y., Peng, X., Shen, T., Clarke, A., Schlegel, C., Martin, P., & Long, G. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-46671-7_2 · Cited by 7
Cong, Y., Chersoni, E., Hsu, Y.-Y., & Blache, P. (2023) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2023.blackboxnlp-1.17 · Cited by 10
Decomposed Prompt Tuning via Low-Rank Reparameterization
Xiao, Y., Xu, L., Li, J., Lu, W., & Li, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.890 · Cited by 18
Calibrated Seq2seq Models for Efficient and Generalizable Ultra-fine Entity Typing
Feng, Y., Pratapa, A., & Mortensen, D. R. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1040 · Cited by 12
Natural Language Decompositions of Implicit Content Enable Better Text Representations
Hoyle, A., Sarkar, R., Goel, P., & Resnik, P. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.815 · Cited by 35
A Demonstration of End-User Code Customization Using Generative AI
Acher, M. (2024) · International Workshop on Variability Modelling of Software-Intensive Systems (VaMoS) · doi:10.1145/3634713.3634732 · Cited by 6
Detecting fake information with knowledge-enhanced AutoPrompt
Che, X., Yang, G., Chen, Y., & Li, Q. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-09491-7 · Cited by 7
PromptSet: A Programmer's Prompting Dataset
Pister, K. S., Paul, D. J., Joshi, I., & Brophy, P. (2024) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1145/3643795.3648395 · Cited by 25
Ma, Y., Luo, S., Shang, Y.-M., Zhang, Y., & Li, Z. (2024) · Scientific Reports · doi:10.1038/s41598-024-69402-7 · Cited by 8
Zagoury, ., Minkov, E., Szpektor, I., & Cohen, W. W. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17681 · Cited by 5
Lexical Knowledge Internalization for Neural Dialog Generation
Wu, Z., Bi, W., Li, X., Kong, L., & Kao, B. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.547 · Cited by 7
Quantum Natural Language Processing: A New and Promising Way to Solve NLP Problems
Bouakba, Y., & Belhadef, H. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-28540-0_17 · Cited by 5
Litvak, M., Rabaev, I., Campos, R., Jorge, A., & Jatowt, A. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591928 · Cited by 5
Making Pretrained Language Models Good Long-tailed Learners
Zhang, C., Ren, L., Wang, J., Wu, W., & Song, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.217 · Cited by 8
Discourse-Aware Prompt for Argument Impact Classification
Chan, C., & Chan, T. (2023) · International Conference on Machine Learning and Computing (ICMLC) · doi:10.1145/3587716.3587743 · Cited by 10
NIR-Prompt: A Multi-task Generalized Neural Information Retrieval Training Framework
Xu, S., Pang, L., Shen, H., & Cheng, X. (2023) · ACM Transactions on Information Systems · doi:10.1145/3626092 · Cited by 6
Xu, S., Li, J., & Xiong, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.226 · Cited by 37
Task-Oriented Multi-Modal Mutual Learning for Vision-Language Models
Long, S., Zhao, Z. W., Yuan, J., Tan, Z., Liu, J., Zhou, L., Wang, S., & Wang, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02007 · Cited by 12
Helping Language Models Learn More: Multi-Dimensional Task Prompt for Few-shot Tuning
Weng, J., Zhang, J., Hu, Y., Fa, D., Xu, X., & Huang, H. (2023) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc53992.2023.10394280 · Cited by 5
What Can a Generative Language Model Answer About a Passage?
Summers-Stay, D. A., Bonial, C., & Voss, C. R. (2021) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/2021.mrqa-1.7 · Cited by 6
Prompt-Based Learning for Aspect-Level Sentiment Classification
Li, G., Lin, F., Chen, W., Dong, D., & Liu, B. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-30111-7_43 · Cited by 9
You Are My Type! Type Embeddings for Pre-trained Language Models
Saeed, M., & Papotti, P. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.336 · Cited by 10
Do Language Models Understand Measurements?
Park, S., Ryu, S., & Choi, E. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.128 · Cited by 19
Generalist Multimodal AI: A Review of Architectures, Challenges and OpportunitiesPreprint
Munikoti, S., Stewart, I., Horawalavithana, S., Kvinge, H., Emerson, T., Thompson, S., & Pazdernik, K. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5732150 · Cited by 6
How Context Affects Language Models' Factual Predictions
Fabio Petroni, P. L. (2020) · Automated Knowledge Base Construction (AKBC) · doi:10.24432/c5201w · Cited by 4
Visual Conceptual Blending with Large-scale Language and Vision Models
Ge, S., & Parikh, D. (2021) · arXiv · doi:10.48550/arxiv.2106.14127 · Cited by 4
“Letting the language model predict masked parts from the prompt then becomes equivalent to the model solving the task of interest (Petroni et al. 2019; Jiang et al. 2020).”
BackgroundMethodTransMap: Pinpointing Mistakes in Neural Code Translation
Wang, B., Li, R., Li, M., & Saxena, P. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616322 · Cited by 16
WhisPAr: Transferring pre-trained audio models to fine-grained classification via Prompt and Adapter
Shi, B., Wang, H., Lu, C., & Zhao, M. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112242 · Cited by 4
Large Language Models With Holistically Thought Could Be Better Doctors
Weng, Y., Li, B., Xia, F., Zhu, M., Sun, B., He, S., Liu, S., Liu, K., Shutao Li, A., & Zhao, J. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9434-8_25 · Cited by 5
Application of Prompt Engineering Techniques to Optimize Information Retrieval in the Metaverse
Karabıyık, M. A., Tan, F. G., & Yüksel, A. S. (2024) · Journal of Metaverse · doi:10.57019/jmv.1543077 · Cited by 4
Agentic Large Language Models for Generating Large-Scale Urban Daily Activity Patterns
Zhang, Y., Zhang, K., Pang, Y., & Sekimoto, Y. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825138 · Cited by 5
Quantum gate control pulse optimization based on the Adam algorithm
Yang, M., Feng, Y., Lu, B., Zhao, H., Ma, G. L., & Wang, L. (2025) · Quantum Information Processing · doi:10.1007/s11128-025-04791-w · Cited by 6
Cross-dialectal Arabic translation: comparative analysis on large language models
Beidas, A., Mohi, K., Ghaddar, F., Ahmad, I., & Abed, S. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1661789 · Cited by 9
Wei, D., Chau, M., & Li, Z. (2025) · MIS Quarterly · doi:10.25300/misq/2025/18416 · Cited by 6
Leveraging Large Language Models for Feature Selection in Drug Recommendation Systems
Dardour, S., & Yanes, N. (2025) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.70370 · Cited by 4
VLN-Game: Vision-Language Equilibrium Search for Zero-Shot Semantic Navigation
Yu, B., Liu, Y., Han, L., Kasaei, H., Li, T., & Cao, M. (2026) · IEEE Transactions on Robotics · doi:10.1109/tro.2026.3677047 · Cited by 24
Multi-Stage Prompt Tuning for Political Perspective Detection in Low-Resource Settings
Kim, K.-M., Lee, M., Won, H.-S., Kim, M., Kim, Y., & Lee, S. (2023) · Applied Sciences · doi:10.3390/app13106252 · Cited by 9
A Prompt Learning Based Intent Recognition Method on a Chinese Implicit Intent Dataset CIID
Liu, S., Li, L., Fang, M., Hung, C., & Yang, S. (2022) · Neural Processing Letters · doi:10.1007/s11063-023-11362-6 · Cited by 5
Ju, T., Zheng, Y., Wang, H., Zhao, H., & Liu, G. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.494 · Cited by 12
Answering Ambiguous Questions via Iterative Prompting
Sun, W., Cai, H., Chen, H., Ren, P., Chen, Z., de Rijke, M., & Ren, Z. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.424 · Cited by 22
FQN Inference in Partial Code by Prompt-tuned Language Model of Code
Huang, Q. Y., Yuan, Z., Xing, Z., Peng, X., Xu, X., & Lu, Q. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3617174 · Cited by 7
Scalable-DSC: A Structural Template Prompt Approach to Scalable Dialogue State Correction
Su, H., Xie, H., Huang, H., Song, S., Fang, R., Huang, X., & Feng, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.490 · Cited by 5
Artificial Intelligence Empowered Language Models: A Review
Negi, A., Verma, C. V., & Tayyebi, Y. (2024) · Lecture notes in networks and systems · doi:10.1007/978-981-99-9524-0_40 · Cited by 5
Dynamic Prompt-Driven Zero-Shot Relation Extraction
Xu, L., Bu, X., & Tian, X. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3402063 · Cited by 5
Parameter-efficient Prompt Learning for 3D Point Cloud Understanding
Sun, H., Wang, Y., Chen, W., Deng, H., & Li, D. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610093 · Cited by 20
Lee, M. (2024) · Fractal and Fractional · doi:10.3390/fractalfract8100552 · Cited by 5
Click Prompt Learning with Optimal Transport for Interactive Segmentation
Liu, J., Wang, H., Yin, W., Sonke, J., & Gavves, E. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73414-4_6 · Cited by 11
He, Y., Feng, J., Ding, G., Guo, Y., & He, T. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129114 · Cited by 4
Shi, Y., & Hu, X. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825237 · Cited by 5
A survey of Few-Shot Relation Extraction combining meta-learning with prompt learning
Hang, T., Wu, W., Feng, J., Djigal, H., & Huang, J. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130534 · Cited by 7
A Disentangled-Attention Based Framework with Persona-Aware Prompt Learning for Dialogue Generation
Liu, P., Huang, Z., Zhang, X., Wang, L., de Melo, G., Lin, X. A., Pang, L., & He, L. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26556 · Cited by 12
Automatic Document Selection for Efficient Encoder Pretraining
Feng, Y., Xia, P., Durme, B. V., & Sedoc, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.647 · Cited by 13
Can Language Models Be Specific? How?
Huang, J., Chang, K. C., Xiong, J., & Hwu, W. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.45 · Cited by 11
RAMP: Retrieval and Attribute-Marking Enhanced Prompting for Attribute-Controlled Translation
Sarti, G., Htut, P. M., Niu, X., Hsu, B., Currey, A., Dinu, G., & Nădejde, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.126 · Cited by 6
Cloze Quality Estimation for Language Assessment
Zhang, Z., Mita, M., & Komachi, M. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.39 · Cited by 4
CTQScorer: Combining Multiple Features for In-context Example Selection for Machine Translation
Aswanth, M., Puduppully, R., Dabre, R., & Kunchukuttan, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.519 · Cited by 24
Application Research of Large Language Models in Medicine: Status, Problems, and Future
Ao, G., Wang, G., Chen, Y., Xia, J., & Su, S. (2023) · International Conference on Information Technology in Medicine and Education · doi:10.1109/itme60234.2023.00151 · Cited by 4
LLM-MHR: A LLM-Augmented Multimodal Hashtag Recommendation Algorithm
Tan, Z., Li, Y., Yuan, X., Meng, S., Li, W., & Mo, T. (2024) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws62655.2024.00100 · Cited by 8
What do Toothbrushes do in the Kitchen? How Transformers Think our World is Structured
Henlein, A., & Mehler, A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.425 · Cited by 8
CoSe-Co: Text Conditioned Generative CommonSense Contextualizer
Bansal, R., Aggarwal, M., Bhatia, S., Kaur, J. N., & Krishnamurthy, B. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.83 · Cited by 6
Decouple knowledge from paramters for plug-and-play language modeling
Cheng, X., Lin, Y., Chen, X., Zhao, D., & Yan, R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.901 · Cited by 21
SMASH: Improving SMAll Language Models’ Few-SHot Ability with Prompt-Based Distillation
Wang, Y., Liu, C., Chen, K., Wang, X., & Zhao, D. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.492 · Cited by 5
IELM: An Open Information Extraction Benchmark for Pre-Trained Language Models
Wang, C., Liu, X., & Song, D. X. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.576 · Cited by 11
Discourse Structure Extraction from Pre-Trained and Fine-Tuned Language Models in Dialogues
Li, C., Huber, P., Wen, X., Amblard, M., Braud, C., & Carenini, G. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.194 · Cited by 23
Measuring the Knowledge Acquisition-Utilization Gap in Pretrained Language Models
Kazemnejad, A., Rezagholizadeh, M., Parthasarathi, P., & Chandar, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.285 · Cited by 10
Language-Agnostic Bias Detection in Language Models with Bias Probing
Köksal, A., Yalcin, O. F., Akbiyik, A., Kılavuz, M. T., Korhonen, A., & Schuetze, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.848 · Cited by 15
PTP: Boosting Stability and Performance of Prompt Tuning with Perturbation-Based Regularizer
Chen, L., Chen, J., Huang, H., & Cheng, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.833 · Cited by 19
Proceedings of the Workshop on Edge and Mobile Foundation Models
(2024) · doi:10.1145/3662006 · Cited by 4
Enriching large language models with semantic lexicons and analogies
Zervakis, G. (2023) · theses.fr (ABES) · doi:10.70675/4f4cd0e5z5987z422az813bzc6e431d1a918 · Cited by 7
Metadata
- Authors
- Jiang, Z., Xu, F. F., Araki, J., & Neubig, G.
- Affiliations
- Carnegie Mellon University (US)
- Robert Bosch (United States) (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Pages
- 423-438
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2991382858
- Cited by
- 1,002
- References
- 52
- Citations by year
- 2026: 76 · 2025: 169 · 2024: 224 · 2023: 285 · 2022: 144 · 2021: 78 · 2020: 25
- FWCI
- 52.35
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1815287)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- prompt generation, ensemble methods
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.