Demystifying Prompts in Language Models via Perplexity Estimation
Open accessAbstract
Language models can be prompted to perform a wide variety of tasks with zero-and few-shot incontext learning.However, performance varies significantly with the choice of prompt, and we do not yet understand why this happens.In this paper, we analyze the factors that contribute to this variance and establish a new empirical hypothesis: the performance of a prompt is predicted by the extent to which the model is familiar with the language it contains.Over a wide range of tasks, we show that the lower the perplexity of the prompt, the better it is able to perform the task, when considering reasonable prompts that are related to it.As part of our analysis, we also devise a method to automatically extend a small seed set of manually written prompts by paraphrasing with GPT3 and backtranslation.This larger set allows us to verify that perplexity is a strong predictor of the success of a prompt and we show that the lowest perplexity prompts are consistently effective.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
This provides a clear signal for detection methods like Perplexity and Min-K% Prob.
“This provides a clear signal for detection methods like Perplexity (Gonen et al., 2023) and Min-K% Prob (Shi et al., 2024), which are built upon this likelihood principle.”
From Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models (2025) · cites this paper as Gonen et al., 2023
The Prompt Accuracy section describes prompt perplexity as a proxy for how expected a prompt is by a model, supporting a connection between perplexity and model likelihood. But the paper uses that measure to study and select prompts; it does not present Perplexity as a detection method or establish a clear signal for detection. No page number is available in our copy. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Character-level Convolutional Networks for Text ClassificationPreprint
Zhang, X., Zhao, J., & LeCun, Y. (2015) · Neural Information Processing Systems · doi:10.48550/arxiv.1509.01626 · Cited by 7,563 · see citing works
DBpedia - A large-scale, multilingual knowledge base extracted from Wikipedia
Lehmann, J., Isele, R., Jakob, M., Jentzsch, A., Kontokostas, D., Mendes, P. N., Hellmann, S., Morsey, M., van Kleef, P., Auer, S., et al. (2015) · Semantic Web · doi:10.3233/sw-140134 · Cited by 3,737 · see citing works
Learning Word Vectors for Sentiment Analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) · Cited by 3,710 · see citing works
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096 · see citing works
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · arXiv · doi:10.48550/arxiv.2104.08691 · Cited by 3,092
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473 · see citing works
No Language Left Behind: Scaling Human-Centered Machine TranslationPreprint
Costa-jussà, M., Cross, J., Çelebi, O., Elbayad, M., Heafield, K., Heffernan, K., Kalbassi, E., Lam, J., Licht, D., Maillard, J., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2207.04672 · Cited by 2,176 · see citing works
5.7 The WORDNET lexical database of English
Ooi, V. B. Y. (2019) · Edinburgh University Press eBooks · doi:10.1515/9781474471459-050 · Cited by 2,034 · see citing works
Making Pre-trained Language Models Better Few-shot Learners
Gao, T., Fisch, A., & Chen, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.295 · Cited by 1,380 · see citing works
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., & Singh, S. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.346 · Cited by 1,279 · see citing works
Neural Network Acceptability Judgments
Warstadt, A., Singh, A., & Bowman, S. R. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00290 · Cited by 984 · see citing works
TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification
Barbieri, F., Camacho-Collados, J., Anke, L. E., & Neves, L. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.148 · Cited by 756 · see citing works
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners
Schick, T., & Schütze, H. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.185 · Cited by 747 · see citing works
Learning How to Ask: Querying LMs with Mixtures of Soft Prompts
Qin, G., & Eisner, J. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.410 · Cited by 342 · see citing works
CARER: Contextualized Affect Representations for Emotion Recognition
Saravia, E., Liu, H.-C. T., Huang, Y.-H., Wu, J., & Chen, Y.-S. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1404 · Cited by 337 · see citing works
RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning
Deng, M., Wang, J., Hsieh, C.-P., Wang, Y., Guo, H. W., Shu, T., Song, M., Xing, E. P., & Hu, Z. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.222 · Cited by 233 · see citing works
Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model
Luccioni, A. S., Viguier, S., & Ligozat, A.-L. (2022) · arXiv · doi:10.48550/arxiv.2211.02001 · Cited by 192 · see citing works
How many data points is a prompt worth?
Le Scao, T., & Rush, A. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.208 · Cited by 179 · see citing works
PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts
Bach, S., Sanh, V., Yong, Z. X., Webson, A., Raffel, C., Nayak, N. V., Sharma, A., Kim, T., Bari, M. S., Fevry, T., et al. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2022.acl-demo.9 · Cited by 178 · see citing works
Large Language Models Struggle to Learn Long-Tail Knowledge
Kandpal, N., Deng, H., Roberts, A., Wallace, E., & Raffel, C. (2022) · arXiv · doi:10.48550/arxiv.2211.08411 · Cited by 143 · see citing works
Reframing Instructional Prompts to GPTk’s Language
Mishra, S. R., Khashabi, D., Baral, C. R., Choi, Y., & Hajishirzi, H. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.50 · Cited by 140 · see citing works
Surface Form Competition: Why the Highest Probability Answer Isn’t Always Right
Holtzman, A., West, P., Shwartz, V., Choi, Y., & Zettlemoyer, L. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.564 · Cited by 127 · see citing works
Self-citationImpact of Pretraining Term Frequencies on Few-Shot Reasoning
Razeghi, Y., Logan, R. L., Gardner, M., & Singh, S. (2022) · arXiv · doi:10.48550/arxiv.2202.07206 · Cited by 53 · see citing works
NorthEuraLex: a wide-coverage lexical database of Northern Eurasia
Dellert, J., Daneyko, T., Münch, A., Ladygina, A., Buch, A., Clarius, N., Grigorjew, I., Balabel, M., Boga, H. I., Baysarova, Z., et al. (2019) · Language Resources and Evaluation · doi:10.1007/s10579-019-09480-6 · Cited by 49 · see citing works
On the Relation between Sensitivity and Accuracy in In-Context Learning
Chen, Y., Zhao, C., Zhou, Y., McKeown, K. R., & He, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.12 · Cited by 37 · see citing works
Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts
Khashabi, D., Lyu, X., Min, S., Qin, L., Richardson, K., Welleck, S., Hajishirzi, H., Khot, T., Sabharwal, A., Singh, S., et al. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.266 · Cited by 29 · see citing works
Multi-Source Social Feedback of Online News FeedsPreprint
Moniz, N., & Torgo, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.07055 · Cited by 22 · see citing works
Toward Human Readable Prompt Tuning: Kubrick’s The Shining is a good movie, and a good prompt too?
Shi, W., Han, X., Gonen, H., Holtzman, A., Tsvetkov, Y., & Zettlemoyer, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.733 · Cited by 18 · see citing works
Self-citationMeasuring Causal Effects of Data Statistics on Language Model's `Factual' PredictionsPreprint
Elazar, Y., Kassner, N., Ravfogel, S., Feder, A., Ravichander, A., Mosbach, M., Belinkov, Y., Schütze, H., & Goldberg, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.14251 · Cited by 15 · see citing works
Han, X., & Tsvetkov, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.12600 · Cited by 10 · see citing works
OPT: Open Pre-trained Transformer Language Models
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.01068
Self-citation
Flagged first, then by citations
Research works printed but not linked: 2
- George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39-41.
- John Wieting, Jonathan Mallinson, and Kevin Gimpel. 2017. Learning paraphrastic sentence embeddings from back-translated bitext. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 274-285.
Cited by
Sorted by citations
A Survey on In-context Learning
Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Chang, B., et al. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.64 · Cited by 649 · see citing works
Security and Privacy Challenges of Large Language Models: A Survey
Das, B. C., Amini, M. H., & Wu, Y. (2025) · ACM Computing Surveys · doi:10.1145/3712001 · Cited by 445 · see citing works
Gao, Y., Li, R., Croxford, E., Caskey, J., Patterson, B. W., Churpek, M. M., Miller, T. A., Dligach, D., & Afshar, M. (2024) · JMIR AI · doi:10.2196/58670 · Cited by 136 · see citing works
State of What Art? A Call for Multi-Prompt LLM Evaluation
Mizrahi, M., Kaplan, G., Malkin, D., Dror, R., Shahaf, D., & Stanovsky, G. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00681 · Cited by 113 · see citing works
Prompting meaning: a hermeneutic approach to optimising prompt engineering with ChatGPT
Henrickson, L., & Meroño-Peñuela, A. (2023) · AI & Society · doi:10.1007/s00146-023-01752-8 · Cited by 136 · see citing works
Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4
Chang, K. K., Cramer, M. H., Soni, S., & Bamman, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.453 · Cited by 110 · see citing works
Jaffal, N. O., Alkhanafseh, M., & Mohaisen, D. (2025) · AI · doi:10.3390/ai6090216 · Cited by 69 · see citing works
The Art of Creative Inquiry—From Question Asking to Prompt Engineering
Sasson-Lazovsky, G., Raz, T., & Kenett, Y. N. (2024) · The Journal of Creative Behavior · doi:10.1002/jocb.671 · Cited by 69 · see citing works
Large Language Model Instruction Following: A Survey of Progresses and Challenges
Lou, R., Zhang, K., & Yin, W. (2023) · Computational Linguistics · doi:10.1162/coli_a_00523 · Cited by 74 · see citing works
Enhancing Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategies
Nan, L., Zhao, Y., Zou, W., Ri, N., Tae, J., Zhang, E., Cohan, A., & Radev, D. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.findings-emnlp.996 · Cited by 88 · see citing works
Yang, L., Wang, Z., Li, Z., Na, J.-C., & Yu, J.-F. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103724 · Cited by 48 · see citing works
Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Models
Zhou, K., Jurafsky, D., & Hashimoto, T. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.335 · Cited by 38 · see citing works
Afshar, M., Resnik, F., Ryan, M. M., Hintzke, J., Lemmon, K. K., Sullivan, A. G., Shah, T., Stordalen, A., Oberst, M., Dambach, J., et al. (2025) · NEJM AI · doi:10.1056/aidbp2401267 · Cited by 40 · see citing works
Prompt Sapper: A LLM-Empowered Production Tool for Building AI Chains
Cheng, Y., Chen, J., Huang, Q. Y., Xing, Z., Xu, X., & Lu, Q. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3638247 · Cited by 35 · see citing works
Language models are not naysayers: an analysis of language models on negation benchmarks
Truong, T. H., Baldwin, T., Verspoor, K. M., & Cohn, T. (2023) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2023.starsem-1.10 · Cited by 45 · see citing works
Language models are not naysayers: an analysis of language models on negation benchmarks
Truong, T. H., Baldwin, T., Verspoor, K., & Cohn, T. (2023) · arXiv · doi:10.48550/arxiv.2306.08189 · Cited by 34
Active Learning Principles for In-Context Learning with Large Language Models
Margatina, K., Schick, T., Αλέτρας, Ν., & Dwivedi-Yu, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.334 · Cited by 42 · see citing works
The language of prompting: What linguistic properties make a prompt successful?
Leidinger, A., van Rooij, R., & Shutova, E. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.618 · Cited by 42 · see citing works
MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learning
Zhang, C., Lin, K., Yang, Z., Wang, J., Li, L., Lin, C.-C., Liu, Z., & Wang, L. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01295 · Cited by 24 · see citing works
Evaluation of African American Language Bias in Natural Language Generation
Deas, N., Grieser, J. A., Kleiner, S., Patton, D. U., Turcan, E., & McKeown, K. R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.421 · Cited by 29 · see citing works
Dr.ICL: Demonstration-Retrieved In-context Learning
Luo, M., Xu, X., Dai, Z., Pasupat, P., Kazemi, M., Baral, C. R., Imbrasaitė, V., & Zhao, V. Y. (2024) · Data Intelligence · doi:10.3724/2096-7004.di.2024.0012 · Cited by 51 · see citing works
Aligning generalization between humans and machines
Ilievski, F., Hammer, B., van Harmelen, F., Paaßen, B., Saralajew, S., Schmid, U., Biehl, M. L., Bolognesi, M., Dong, X., Gashteovski, K., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01109-4 · Cited by 23 · see citing works
Large Language Models and Medical Knowledge Grounding for Diagnosis PredictionPreprint
Gao, Y., Li, R., Croxford, E., Tesch, S., To, D., Caskey, J., Patterson, B. W., Churpek, M. M., Miller, T. A., Dligach, D., et al. (2023) · medRxiv · doi:10.1101/2023.11.24.23298641 · Cited by 21 · see citing works
Calle, P., Shao, R., Liu, Y., Hébert, E. T., Kendzor, D. E., Neil, J. M., Businelle, M. S., & Pan, C. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641965 · Cited by 18 · see citing works
Instruct Me More! Random Prompting for Visual In-Context Learning
Zhang, J., Wang, B.-W., Li, L.-Z., Nakashima, Y., & Nagahara, H. (2023) · IEEE Workshop/Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00258 · Cited by 42 · see citing works
Morris, M. R. (2024) · Communications of the ACM · doi:10.1145/3673861 · Cited by 17 · see citing works
PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models
Zou, W., Geng, R., Wang, B., & Jia, J. (2024) · arXiv · doi:10.48550/arxiv.2402.07867 · Cited by 16 · see citing works
Data Curation Alone Can Stabilize In-context Learning
Chang, T.-Y., & Jia, R. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2023.acl-long.452 · Cited by 76 · see citing works
CoLE: A collaborative legal expert prompting framework for large language models in law
Li, B., Fan, S., Zhu, S., & Wen, L.-J. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113052 · Cited by 14 · see citing works
Hidayatullah, A. F., Apong, R. A. A. H. M., Lai, D. T. C., & Qazi, A. (2025) · Social Network Analysis and Mining · doi:10.1007/s13278-025-01444-9 · Cited by 14 · see citing works
GENERATIVE PRE-TRAINED TRANSFORMER 3
Golubenko, O., & Pidmohylʹnyy, O. O. (2022) · ITSynergy · doi:10.53920/its-2022-2-2 · Cited by 15 · see citing works
Toward Human Readable Prompt Tuning: Kubrick’s The Shining is a good movie, and a good prompt too?
Shi, W., Han, X., Gonen, H., Holtzman, A., Tsvetkov, Y., & Zettlemoyer, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.733 · Cited by 18 · see citing works
Self-citationImpact of Sample Selection on In-context Learning for Entity Extraction from Scientific Writing
Bölücü, N., Rybinski, M., & Wan, S. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.findings-emnlp.338 · Cited by 19 · see citing works
Can deepfakes be used to study emotion perception? A comparison of dynamic face stimuli
Becker, C., Conduit, R., Chouinard, P. A., & Laycock, R. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02443-y · Cited by 13 · see citing works
Hou, R., Wang, X., Zhang, W.-Y., Song, Z., Wang, K., Chen, Y., Liu, J.-P., & Ruan, T. (2025) · IEEE journal of biomedical and health informatics · doi:10.1109/jbhi.2025.3529682 · Cited by 16 · see citing works
Pecher, B., Srba, I., & Bieliková, M. (2024) · ACM Computing Surveys · doi:10.1145/3691339 · Cited by 15 · see citing works
Navigating the Risks of Using Large Language Models for Text Annotation in Social Science Research
Lin, H., & Zhang, Y. (2025) · Social Science Computer Review · doi:10.1177/08944393251366243 · Cited by 17 · see citing works
“AI, will you help?” How learners use Artificial Intelligence when writing
Krajka, J., & Olszak, I. (2024) · XLinguae · doi:10.18355/xl.2024.17.01.03 · Cited by 11 · see citing works
Xu, Z., & Wall, E. (2024) · Visualization · doi:10.1109/vis55277.2024.00033 · Cited by 12 · see citing works
Performance of Large Language Models in a Computer Science Degree Program
Krüger, T., & Gref, M. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-50485-3_40 · Cited by 10 · see citing works
Adaptive Prompt Routing for Arbitrary Text Style Transfer with Pre-trained Language Models
Liu, Q., Qin, J.-H., Ye, W.-X., Mou, H., He, Y., & Wang, K.-Z. (2024) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29832 · Cited by 10 · see citing works
Self-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrations
Chen, W., Wu, C.-K., Chen, Y.-N., & Chen, H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.968 · Cited by 9 · see citing works
Croxford, E., Gao, Y., Patterson, B. W., To, D., Tesch, S., Dligach, D., Mayampurath, A. M., Churpek, M. M., & Afshar, M. (2024) · medRxiv · doi:10.1101/2024.03.20.24304620 · Cited by 8 · see citing works
Security and Privacy of Large Language Models: Threat Taxonomy, Ethical Implications, and Governance
Pribisalić, M., & Martinčić-Ipšić, S. (2026) · AI · doi:10.3390/ai7050152 · Cited by 9 · see citing works
Metric-Based In-context Learning: A Case Study in Text Simplification
Vadlamannati, S., & Şahin, G. G. (2023) · International Conference on Natural Language Generation · doi:10.18653/v1/2023.inlg-main.18 · Cited by 7 · see citing works
Explanation Selection Using Unlabeled Data for Chain-of-Thought Prompting
Ye, X., & Durrett, G. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.41 · Cited by 14 · see citing works
Improving Probability-based Prompt Selection Through Unified Evaluation and Analysis
Yang, S., Kim, J., Jang, J., Ye, S., Lee, H., & Seo, M. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00666 · Cited by 15 · see citing works
Data Similarity is Not Enough to Explain Language Model Performance
Yauney, G., Reif, E., & Mimno, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.695 · Cited by 6 · see citing works
Security and privacy in LLMs: A comprehensive survey of threats and mitigation strategies
Berini, A. D. E., Jamil, N., Benrazek, A., Lakas, A., Ismail, L., Ferrag, M. A., & Lam, K. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104241 · Cited by 9 · see citing works
Yang, Z., Zhang, Y., Sui, D., Liu, C., Zhao, J., & Liu, K. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.331 · Cited by 45 · see citing works
LLM-driven Instruction Following: Progresses and Concerns
Yin, W., Ye, Q., Liu, P., Ren, X., & Schütze, H. (2023) · doi:10.18653/v1/2023.emnlp-tutorial.4 · Cited by 7 · see citing works
Bridging Information-Theoretic and Geometric Compression in Language Models
Cheng, E., Kervadec, C., & Baroni, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.762 · Cited by 10 · see citing works
GenKP: generative knowledge prompts for enhancing large language models
Li, X., Peng, S., Yada, S., Wakamiya, S., & Aramaki, E. (2025) · Applied Intelligence · doi:10.1007/s10489-025-06318-3 · Cited by 5 · see citing works
Context Example Selection for LLM Generated Relevance Assessments
McKechnie, J., McDonald, G., & Macdonald, C. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88708-6_19 · Cited by 5 · see citing works
Hint-Enhanced In-Context Learning Wakes Large Language Models Up For Knowledge-Intensive Tasks
Wang, Y., Guo, Q., Ni, X., Shi, C., Liu, L., Jiang, H., & Yang, Y. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447527 · Cited by 6 · see citing works
Large Language Models for Recommender Systems: A Problem-Driven Survey
Guan, Z., Zhong, W., Liu, W., Liu, Y., Kou, H., Chi, X., & Zhao, X. (2025) · Tsinghua Science & Technology · doi:10.26599/tst.2025.9010137 · Cited by 5 · see citing works
Li, B., Liu, T., Ma, J., Dai, J., & Liao, P. (2025) · Computer Networks · doi:10.1016/j.comnet.2025.111074 · Cited by 4 · see citing works
In-Context Demonstration Selection with Cross Entropy Difference
Iter, D., Pryzant, R., Xu, R., Wang, S., Liu, Y., Yi‐chong, X., & Zhu, C. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.81 · Cited by 8 · see citing works
CTQScorer: Combining Multiple Features for In-context Example Selection for Machine Translation
Aswanth, M., Puduppully, R., Dabre, R., & Kunchukuttan, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.519 · Cited by 4 · see citing works
Mo, Y., Tang, M., Lin, R., Zhou, B., & Li, X. (2025) · IEEE Access · doi:10.1109/access.2025.3582519 · Cited by 3 · see citing works
Language Model Guided Reinforcement Learning in Quantitative Trading
Darmanin, A., & Vella, V. (2025) · International Conference on Foundation and Large Language Models (FLLM) · doi:10.1109/fllm67465.2025.11390898 · Cited by 3 · see citing works
Yang, Z., Dai, D., Wang, P., & Sui, Z. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.880 · Cited by 3 · see citing works
EEG correlates of static and dynamic face perception: The role of naturalistic motion
Becker, C., Conduit, R., Chouinard, P. A., & Laycock, R. (2024) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2024.108986 · Cited by 3 · see citing works
What Is Wrong with My Model? Identifying Systematic Problems with Semantic Data Slicing
Yang, C., Hong, Y., Lewis, G. A., Wu, T.-S., & Kästner, C. (2024) · International Conference on Automated Software Engineering · doi:10.1145/3691620.3695033 · Cited by 5 · see citing works
E-InMeMo: Enhanced Prompting for Visual In-Context Learning
Zhang, J., Wang, B.-W., Liu, H., Li, L.-Z., Nakashima, Y., & Nagahara, H. (2025) · Journal of Imaging · doi:10.3390/jimaging11070232 · Cited by 5 · see citing works
Engagement-Driven Content Generation with Large Language Models
Coppolillo, E., Cinus, F., Minici, M., Bonchi, F., & Manco, G. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3736932 · Cited by 3 · see citing works
Frugal Prompting for Dialog Models
Santra, B., Basak, S., De, A., Gupta, M., & Goyal, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.290 · Cited by 3 · see citing works
Robustness of Named-Entity Replacements for In-Context Learning
Goodarzi, S., Kagita, N., Minn, D., Wang, S., Dessì, R., Toshniwal, S., Williams, A., Lanchantin, J., & Sinha, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.728 · Cited by 3 · see citing works
Input-oriented demonstration learning for hybrid evidence fact verification
Chen, C., Chen, W., Zheng, J., Luo, A., Cai, F., & Zhang, Y. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123191 · Cited by 5 · see citing works
Afshar, M., Gao, Y., Wills, G. J., Wang, J., Churpek, M. M., Westenberger, C. J., Kunstman, D., Gordon, J. E., Liao, F., & Patterson, B. W. (2024) · medRxiv · doi:10.1101/2024.01.23.24301692 · Cited by 3 · see citing works
Systematic survey of various prompt optimization methods and their classifications
Jain, A. M., & Jindal, M. (2025) · International Conference on Computing and Artificial Intelligence (ICCAI) · doi:10.1109/iccai66501.2025.00085 · Cited by 2 · see citing works
Mobadersani, M., Candaş, A. B., Kuruoğlu, M., & Tokdemir, O. B. (2025) · Buildings · doi:10.3390/buildings16010120 · Cited by 2 · see citing works
Gupta, P., Jain, N., & Bhat, A. (2024) · International Conference on Applied Artificial Intelligence and Computing (ICAAIC) · doi:10.1109/icaaic60222.2024.10575336 · Cited by 2 · see citing works
LLM Based Physical Verification Runset Generator
Francisco, L., & Arikati, S. (2024) · ACM/IEEE Workshop on Machine Learning for CAD (MLCAD) · doi:10.1145/3670474.3685976 · Cited by 2 · see citing works
Tiwari, A., Kadian, K., Garg, N., Kaur, S., & Dwivedi, V. K. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-75164-6_4 · Cited by 2 · see citing works
ContextualCoder: Adaptive In-Context Prompting for Programmatic Visual Question Answering
Shen, R., Inoue, N., Guan, D., Cai, R., Kot, A. C., & Shinoda, K. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3543043 · Cited by 1 · see citing works
Pragmatic inference and mapping for conversational implicature
Hu, J., Liu, Y., Ma, L., Zhang, W.-N., & Liu, T. (2025) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-025-02741-7 · Cited by 2 · see citing works
Zhang, J., Wada, S., Sugimoto, K., Niitsu, T., Fukushima, K., Kida, H., Wang, B., Konishi, S., Okada, K., Nakashima, Y., et al. (2025) · Journal of Imaging · doi:10.3390/jimaging11100353 · Cited by 1 · see citing works
Hu, J., Bi, W., Zhou, Y., Zheng, J., Zhang, S., Chu, H., Wang, L., & Li, Q. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.130118 · Cited by 1 · see citing works
TaskEval: Assessing Difficulty of Code Generation Tasks for Large Language Models
Tambon, F., Nikanjam, A., Zid, C., Khomh, F., & Antoniol, G. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3773285 · Cited by 2 · see citing works
TACOS: Open Tagging and Comparative Scoring for Instruction Fine-Tuning Data Selection
He, X., Yu, H., Sun, Q., Cheng, A., Zhang, T., Liu, C., & Guo, S. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11209076 · Cited by 1 · see citing works
PRSM: A Measure to Evaluate CLIP’s Robustness Against Paraphrases
Schlegel, U., Weeber, F., Lan, J. W., & Seidl, T. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-6963-2_6 · Cited by 1 · see citing works
Aiyappa, R., Senthilmani, S., An, J., Kwak, H., & Ahn, Y. (2026) · PeerJ Computer Science · doi:10.7717/peerj-cs.3540 · Cited by 7 · see citing works
Event-Centric Opinion Mining via In-Context Learning with ChatGPT
Wu, L., Chen, Y., Yang, J., Shi, G., Qi, X., & Deng, S. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-7224-1_7 · Cited by 1 · see citing works
Unnatural language processing: How do language models handle machine-generated prompts?
Kervadec, C., Franzon, F., & Baroni, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.959 · Cited by 1 · see citing works
ReadPrompt: A Readable Prompting Method for Reliable Knowledge Probing
Wang, Z., Ye, L., Wang, H., Kwan, W.-C., Ho, D., & Wong, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.501 · Cited by 1 · see citing works
Open Llama2 Models for the Lithuanian Language
Nakvosas, A., Daniušis, P., & Mulevičius, V. (2025) · Informatica · doi:10.15388/25-infor592 · Cited by 1 · see citing works
LLM Based Physical Verification Runset Generator
Francisco, L., & Arikati, S. (2024) · ACM/IEEE Workshop on Machine Learning for CAD (MLCAD) · doi:10.1109/mlcad62225.2024.10740218 · Cited by 1 · see citing works
Du, Y., Zhang, Q., Fu, S., Hou, Y., & Han, H. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-07214-w · Cited by 1 · see citing works
Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models
Tao, Y., Wang, T., Dong, Y., Liu, H., Zhang, K., Hu, X., & Li, G. (2025) · arXiv · doi:10.48550/arxiv.2510.09259 · Cited by 1 · see citing works
Leveraging Multiple Teachers for Test-Time Adaptation of Language-Guided Classifiers
Wei, K., Ghosh, S., Menon, R. R., & Srivastava, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.471
Improving Input-label Mapping with Demonstration Replay for In-context Learning
Gong, Z., Liu, J., Wang, Q., Wang, J., Cai, X., Zhao, D., & Yan, R. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.995
SMOOTHIE: Label Free Language Model RoutingPreprint
Guha, N., Chen, M. F., Chow, T., Khare, I. S., & Ré, C. (2025) · Qeios · doi:10.32388/f319w4
Probability-Selected Demonstrations for Enhanced Zero-Shot in-Context Learning
Bai, B. (2024) · International Conference on Artificial Intelligence and Computer Engineering (ICAICE) · doi:10.1109/icaice63571.2024.10864087
Multi-Modal Relation Extraction Based on Shortest Dependency Path and Large Language ModelsPreprint
Xue, H., Dai, Y., Ma, J., & Guo, X. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5131704
Leveraging Correctness-Based Learning for Efficient Reasoning via Automated Process Supervision
Yao, Y., Liu, M., Deng, G., Liu, Z., Wu, D. O., Guo, Z., Wu, H., & Song, L. (2025) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2025.3581101
Is a Similar Task Useful for Few-Shot Selection? Aspect Term Extraction Using LLM
Imazato, K., & Shimada, K. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97144-0_5
Influencia del formato y la complejidad de prompts en calidad de respuestas de modelos generativos
Vera, W. F. Z., & Carrillo, J. J. M. (2025) · Revista Científica de Informática ENCRIPTAR · doi:10.56124/encriptar.v8i16.011
Qiu, X., Cheng, L., Hao, T., & Tan, X. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3630623
- Authors
- Gonen, H., Iyer, S., Blevins, T., Smith, N., & Zettlemoyer, L.
- Affiliations
- University of Washington (US)
- Allen Institute for Artificial Intelligence (US)
- Venue
- Findings of the Association for Computational Linguistics: EMNLP 2023
- Published
- 2023
- Pages
- 10136-10148
- OpenAlex
- W4389520255
- Cited by
- 109
- References
- 32
- Citations by year
- 2026: 14 · 2025: 33 · 2024: 30 · 2023: 31 · 2022: 1
- FWCI
- 19.14
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- prompt engineering, language models, in-context learning, zero-shot learning, few-shot learning, paraphrase, back-translation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review