CoChat
Sign Up
PreprintCited by 26Topic ModelingNatural Language Processing TechniquesInformation Retrieval and Search Behavior

A Systematic Survey of Automatic Prompt Optimization Techniques

Open access
Ramnath, Kiran, Zhou, Kang, Guan, Sheng, Mishra, Soumya Smruti, Qi, Xuan, Shen, Zhengyuan, Wang, Shuai, Woo, Sangmin +13 more·2025·arXiv
DOI 10.48550/arxiv.2502.16923arXiv 2502.16923

Preprint — published as 10.18653/v1/2025.emnlp-main.1681

Open this paper in CoChat — free accountOpen PDF

Abstract

Since the advent of large language models (LLMs), prompt engineering has been a crucial step for eliciting desired responses for various Natural Language Processing (NLP) tasks. However, prompt engineering remains an impediment for end users due to rapid advances in models, tasks, and associated best practices. To mitigate this, Automatic Prompt Optimization (APO) techniques have recently emerged that use various automated techniques to help improve the performance of LLMs on various tasks. In this paper, we present a comprehensive survey summarizing the current progress and remaining challenges in this field. We provide a formal definition of APO, a 5-part unifying framework, and then proceed to rigorously categorize all relevant works based on their salient features therein. We hope to spur further research guided by our framework.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2502.16923

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Prompt Optimization enables LLM agents to self-evolve by refining the instructions fed to the backbone model, directly altering the model's behavior without modifying model weights.

    “While memory evolution focused on what knowledge an agent retains, Prompt Optimization (PO) enables LLM agents to self-evolve by refining the instructions it feeds to the backbone model, which directly alters the model's behavior without modifying model weights(Ramnath et al., 2025).”

    From A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence (2025) · cites this paper as Ramnath et al., 2025

    The Introduction (p. unavailable) says black-box automatic prompt optimization improves task performance through automated prompt improvements, and §3.1 shows prompts being evaluated as inputs to a target model. This supports refining instructions to change model outputs without changing the target model’s weights. The survey does not establish the broader claim that this enables LLM agents to self-evolve. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 138

  1. Fernando Alva-Manchego, Louis Martin, Antoine Bor- des, Carolina Scarton, Benoît Sagot, and Lucia Specia. 2020. Asset: A dataset for tuning and evaluation of sentence simplification models with multiple rewriting transformations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguis- tics, pages 4668-4679. · DOI
  2. Afra Amini, Tim Vieira, and Ryan Cotterell. 2024. Di- rect preference optimization with an offset. In Findings of the Association for Computational Linguistics: ACL 2024, pages 9954-9972, Bangkok, Thailand. Associa- tion for Computational Linguistics. · DOI
  3. R. Anantha, Svitlana Vakulenko, Zhucheng Tu, S. Long- pre, Stephen G. Pulman, and Srinivas Chappidi. 2020. Open-domain question answering goes conversational via question rewriting. In North American Chapter of the Association for Computational Linguistics.
  4. Jacob Andreas, Johannes Bufe, David Burkett, Charles C. Chen, Joshua Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eis- ner, Hao Fang, Alan Guo, David Leo Wright Hall, Kristin Delia Hayes, Kellie Hill, Diana Ho, Wendy Iwaszuk, Smriti Jha, Dan Klein, Jayant Krishnamurthy, Theo Lanman, Percy Liang, C. H. Lin, Ilya Lintsbakh, Andy McGovern, Aleksandr Nisnevich, Adam Pauls, Dmitrij Petters, Brent Read, Dan Roth, Subhro Roy, Jesse Rusak, Beth Ann Short, Div Slomin, B Snyder, Stephon Striplin, Yu Su, Zachary Tellman, Sam Thom- son, A. A. Vorobev, Izabela Witoszko, Jason Wolfe, A. G. Wray, Yuchen Zhang, and Alexander Zotov. 2020. Task-oriented dialogue as dataflow synthesis. Transac- tions of the Association for Computational Linguistics, 8:556-571.
  5. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2019. Piqa: Reasoning about physical commonsense in natural language. In AAAI Conference on Artificial Intelligence.
  6. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326. · arXiv
  7. Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018. Multiwoz-a large-scale multi- domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5026.
  8. Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Niehues Jan, Stüker Sebastian, Sudoh Katsuitho, Yoshino Koichiro, and Federmann Christian. 2017. Overview of the iwslt 2017 evaluation campaign. In In- ternational Workshop on Spoken Language Translation.
  9. Yongchao Chen, Jacob Arkin, Yilun Hao, Yang Zhang, Nicholas Roy, and Chuchu Fan. 2024. PRompt opti- mization in multi-step tasks (PROMST): Integrating human feedback and heuristic-based sampling. In Pro- ceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing, pages 3859-3920, Miami, Florida, USA. Association for Computational Linguistics. · DOI
  10. Jiale Cheng, Xiao Liu, Kehan Zheng, Pei Ke, Hongn- ing Wang, Yuxiao Dong, Jie Tang, and Minlie Huang. 2024. Black-box prompt optimization: Aligning large language models without model training. In Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3201-3219, Bangkok, Thailand. Association for Computational Linguistics. · DOI
  11. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt- 4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6. · DOI
  12. Minje Choi, Jiaxin Pei, Sagar Kumar, Chang Shu, and David Jurgens. 2023. Do llms understand social knowl- edge? evaluating the sociability of large language mod- els with socket benchmark. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 11370-11403.
  13. Christopher Cieri, Mark Liberman, Sunghye Cho, Stephanie Strassel, James Fiumara, and Jonathan Wright. 2022. Reflections on 30 years of language re- source development and sharing. In Proceedings of the Thirteenth Language Resources and Evaluation Con- ference, pages 543-550, Marseille, France. European Language Resources Association.
  14. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question an- swering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457. · arXiv
  15. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plap- pert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. · arXiv
  16. Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introduc- ing the world's first truly open instruction-tuned llm. Company Blog of Databricks. · DOI
  17. Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang, and Ming Zhou. 2020. Mutual: A dataset for multi-turn dialogue reasoning. ArXiv, abs/2004.04494. · arXiv
  18. Franck Dernoncourt and Ji Young Lee. 2017. Pubmed 200k rct: a dataset for sequential sentence classification in medical abstracts. In International Joint Conference on Natural Language Processing.
  19. Robert C. Detrano, András Jánosi, Walter Steinbrunn, Matthias Emil Pfisterer, Johann-Jakob Schmid, Sarbjit Sandhu, Kern Guppy, Stella Lee, and Victor Froelicher. 1989. International application of a new probability algorithm for the diagnosis of coronary artery disease. The American journal of cardiology, 64 5:304-10.
  20. Shizhe Diao, Zhichao Huang, Ruijia Xu, Xuechun Li, Yong Lin, Xiao Zhou, and Tong Zhang. 2022. Black- box prompt learning for pre-trained language models. arXiv preprint arXiv:2201.08531. · DOI
  21. Rezarta Islamaj Dogan, Robert Leaman, and Zhiyong Lu. 2014. Ncbi disease corpus: a resource for disease name recognition and concept normalization. Journal of biomedical informatics, 47:1-10. · DOI
  22. William B. Dolan and Chris Brockett. 2005. Automat- ically constructing a corpus of sentential paraphrases. In International Joint Conference on Natural Language Processing.
  23. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. 2024a. A survey on in-context learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1107-1128, Miami, Florida, USA. Association for Computational Linguistics. · DOI · Paper page
  24. Yihong Dong, Kangcheng Luo, Xue Jiang, Zhi Jin, and Ge Li. 2024b. PACE: Improving prompt with actor- critic editing for large language model. In Findings of the Association for Computational Linguistics: ACL 2024, pages 7304-7323, Bangkok, Thailand. Associa- tion for Computational Linguistics. · DOI
  25. Yingjun Du, Wenfang Sun, and Cees GM Snoek. 2024. Ipo: Interpretable prompt optimization for vision- language models. arXiv preprint arXiv:2410.15397. · arXiv
  26. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In North American Chapter of the Association for Computational Linguis- tics.
  27. Stefan Daniel Dumitrescu, Petru Rebeja, Beáta Lőrincz, Mihaela Gȃman, Mihai Daniel Ilie, Andrei Pruteanu, Adriana Stan, Luciana Morogan, Traian Rebedea, and Sebastian Ruder. 2021. Liro: Benchmark and leader- board for romanian language tasks. In NeurIPS Datasets and Benchmarks.
  28. Ibrahim Abu Farha and Walid Magdy. 2020b. From arabic sentiment analysis to sarcasm detection: The arsarcasm dataset. In OSACT. Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rock- täschel. 2023. Promptbreeder: Self-referential self-improvement via prompt evolution. ArXiv, abs/2309.16797. · arXiv
  29. Rory A. Fisher. 1936. The use of multiple measure- ments in taxonomic problems. Annals of Human Genet- ics, 7:179-188.
  30. Noa Garcia, Chentao Ye, Zihua Liu, Qingtao Hu, Mayu Otani, Chenhui Chu, Yuta Nakashima, and Teruko Mita- mura. 2020. A dataset and baselines for visual question answering on art. In European Conference on Computer Vision, pages 92-108. · DOI
  31. Miguel Garc'ia-Orteg'on, Gregor N. C. Simm, Austin Tripp, José Miguel Hernández-Lobato, Andreas Ben- der, and Sergio Bacallado. 2021. Dockstring: Easy molecular docking yields better benchmarks for ligand design. Journal of Chemical Information and Modeling, 62:3486 -3502.
  32. Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. Creating training corpora for nlg micro-planners. In Annual Meeting of the Association for Computational Linguistics.
  33. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346-361.
  34. Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. Samsum corpus: A human- annotated dialogue dataset for abstractive summariza- tion. In Proceedings of the 2nd Workshop on New Fron- tiers in Summarization, pages 70-79. · DOI
  35. Chulaka Gunasekara, Jonathan K. Kummerfeld, Lazaros Polymenakos, and Walter S. Lasecki. 2019. Dstc7 task 1: Noetic end-to-end response selection. Proceedings of the First Workshop on NLP for Conversational AI.
  36. Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. 2024. Connecting large language models with evolu- tionary algorithms yields powerful prompt optimizers. In The Twelfth International Conference on Learning Representations.
  37. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Ja- cob Steinhardt. 2020. Measuring massive multitask language understanding. ArXiv, abs/2009.03300. · arXiv
  38. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving Cho-Jui Hsieh, Si Si, Felix Yu, and Inderjit Dhillon. 2024. Automatic engineering of long prompts. In Find- ings of the Association for Computational Linguistics: ACL 2024, page 10672-10685, Bangkok, Thailand. Association for Computational Linguistics. · DOI
  39. Minqing Hu and Bing Liu. 2004. Mining and sum- marizing customer reviews. Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining.
  40. Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos qa: Machine reading com- prehension with contextual commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2391-2401.
  41. Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. 2023. Make-an-audio: Text-to- audio generation with prompt-enhanced diffusion mod- els. In International Conference on Machine Learning, pages 13916-13932. · DOI
  42. PMLR.
  43. Yatai Ji, Jiacheng Zhang, Jie Wu, Shilong Zhang, Shoufa Chen, Chongjian GE, Peize Sun, Weifeng Chen, Wenqi Shao, Xuefeng Xiao, et al. 2024. Prompt-a-video: Prompt your video diffusion model via preference- aligned llm. arXiv preprint arXiv:2412.15156. · arXiv
  44. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2020. What disease does this patient have? a large-scale open domain ques- tion answering dataset from medical exams. ArXiv, abs/2009.13081. · arXiv
  45. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. In Pro- ceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Process- ing (EMNLP-IJCNLP), pages 2567-2577. · DOI
  46. Hideaki Joko, Shubham Chatterjee, Andrew Ramsay, Arjen P De Vries, Jeff Dalton, and Faegheh Hasibi. 2024. Doing personal laps: Llm-augmented dialogue construction for personalized multi-session conversa- tional search. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 796-806.
  47. Gurusha Juneja, Nagarajan Natarajan, Hua Li, Jian Jiao, and Amit Sharma. 2024. Task facet learning: A struc- tured approach to prompt optimization. arXiv preprint arXiv:2406.10504. · arXiv
  48. David Jurgens, Srijan Kumar, Raine Hoover, Daniel A. McFarland, and Dan Jurafsky. 2018. Measuring the evolution of a scientific field through citation frames. Transactions of the Association for Computational Lin- guistics, 6:391-406.
  49. Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2022. Demonstrate-search-predict: Com- posing retrieval and language models for knowledge- intensive nlp. arXiv preprint arXiv:2212.14024. · DOI
  50. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christo- pher Potts. 2024. Dspy: Compiling declarative language model calls into self-improving pipelines. · DOI
  51. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466. · DOI
  52. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading com- prehension dataset from examinations. arXiv preprint arXiv:1704.04683. · arXiv
  53. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open do- main question answering. ArXiv, abs/1906.00300. · arXiv
  54. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tun- ing. In Proceedings of the 2021 Conference on Empir- ical Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Dominican Repub- lic. Association for Computational Linguistics. · DOI · Paper page
  55. Hector J. Levesque, Ernest Davis, and L. Morgenstern. 2011. The winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning.
  56. Cheng Li, Jindong Wang, Yixuan Zhang, Kaijie Zhu, Wenxin Hou, Jianxun Lian, Fang Luo, Qiang Yang, and Xing Xie. 2023b. Large language models under- stand and can be enhanced by emotional stimuli. arXiv preprint arXiv:2307.11760. · arXiv
  57. Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. 2010. A contextual-bandit approach to per- sonalized news article recommendation. In Proceedings of the 19th International Conference on World Wide Web, WWW '10, page 661-670, New York, NY, USA. Association for Computing Machinery. · DOI · Paper page
  58. Moxin Li, Wenjie Wang, Fuli Feng, Yixin Cao, Jizhi Zhang, and Tat-Seng Chua. 2023c. Robust prompt opti- mization for large language models against distribution shifts. In Proceedings of the 2023 Conference on Em- pirical Methods in Natural Language Processing, pages 1539-1554, Singapore. Association for Computational Linguistics. · DOI
  59. Zekun Li, Baolin Peng, Pengcheng He, Michel Galley, Jianfeng Gao, and Xifeng Yan. 2023d. Guiding large language models via directional stimulus prompting. arXiv preprint arXiv:2302.11520. · arXiv
  60. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human false- hoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214-3252. · DOI
  61. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, Septem- ber 6-12, 2014, Proceedings, Part V 13, pages 740-755. · DOI · Paper page
  62. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blun- som. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Annual Meeting of the Association for Computational Linguistics.
  63. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paran- jape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024a. Lost in the middle: How language mod- els use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173. · DOI · Paper page
  64. Shengcai Liu, Caishun Chen, Xinghua Qu, Ke Tang, and Yew Soon Ong. 2023. Large language models as evolutionary optimizers. 2024 IEEE Congress on Evo- lutionary Computation (CEC), pages 1-8.
  65. Shihong Liu, Samuel Yu, Zhiqiu Lin, Deepak Pathak, and Deva Ramanan. 2024b. Language models as black- box optimizers for vision-language models. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12687-12697. · DOI
  66. Yilun Liu, Minggui He, Feiyu Yao, Yuhe Ji, Shimin Tao, Jingzhou Du, Duan Li, Jian Gao, Li Zhang, Hao Yang, et al. 2024d. What do you want? user-centric prompt generation for text-to-image synthesis via multi-turn guidance. arXiv preprint arXiv:2408.12910. · arXiv
  67. Xuan Do Long, Yiran Zhao, Hannah Brown, Yuxi Xie, James Xu Zhao, Nancy F. Chen, Kenji Kawaguchi, Michael Shieh, and Junxian He. 2024. Prompt opti- mization via adversarial in-context learning. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7308-7327, Bangkok, Thailand. Association for Computational Linguistics. · DOI
  68. Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. In SIGDIAL Conference.
  69. Junru Lu, Siyu An, Min Zhang, Yulan He, Di Yin, and Xing Sun. 2025. FIPO: Free-form instruction-oriented prompt optimization with preference dataset and mod- ular fine-tuning schema. In Proceedings of the 31st International Conference on Computational Linguistics, page 11029-11047, Abu Dhabi, UAE. Association for Computational Linguistics.
  70. Yao Lu, Jiayi Wang, Raphael Tang, Sebastian Riedel, and Pontus Stenetorp. 2024. Strings from the library of babel: Random sampling as a strong baseline for prompt optimisation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies (Volume 1: Long Papers), page 2221-2231, Mexico City, Mexico. Association for Computational Linguis- tics. · DOI
  71. Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, re- lations, and coreference for scientific knowledge graph construction. ArXiv, abs/1808.09602. · arXiv
  72. Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Pro- ceedings of the 49th annual meeting of the association for computational linguistics: Human language tech- nologies, pages 142-150.
  73. Oscar Mañas, Pietro Astolfi, Melissa Hall, Can- dace Ross, Jack Urbanek, Adina Williams, Aish- warya Agrawal, Adriana Romero-Soriano, and Michal Drozdzal. 2024. Improving text-to-image consistency via automatic prompt optimization. arXiv preprint arXiv:2403.17804. · arXiv
  74. Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language de- cathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730. · DOI
  75. Rimon Melamed, Lucas H. McCabe, Tanay Wakhare, Yejin Kim, H. Howie Huang, and Enric Boix-Adsera. 2024. Prompts have evil twins. M Jehanzeb Mirza, Mengjie Zhao, Zhuoyuan Mao, Sivan Doveh, Wei Lin, Paul Gavrikov, Michael Dorken- wald, Shiqi Yang, Saurav Jha, Hiromi Wakaki, et al. 2024. Glov: Guided large language models as implicit optimizers for vision language models. arXiv preprint arXiv:2410.06154. · DOI
  76. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2021. Cross-task generalization via natural language crowdsourcing instructions. In Annual Meeting of the Association for Computational Linguistics.
  77. Ioannis Mollas, Zoe Chrysopoulou, Stamatis Karlos, and Grigorios Tsoumakas. 2020. Ethos: an on- line hate speech detection dataset. arXiv preprint arXiv:2006.08328. · DOI
  78. Ramesh Nallapati, Bowen Zhou, Cícero Nogueira dos Santos, Çaglar Gülçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to- sequence rnns and beyond. In Conference on Com- putational Natural Language Learning.
  79. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. ArXiv, abs/1808.08745. · arXiv
  80. Ehsan Nezhadarya, Yang Liu, and Bingbing Liu. 2019. Boxnet: A deep learning method for 2d bounding box estimation from bird's-eye view point cloud. In 2019 IEEE Intelligent Vehicles Symposium (IV), pages 1557- 1564. IEEE.
  81. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2019. Adversarial nli: A new benchmark for natural language understanding. ArXiv, abs/1910.14599. · arXiv
  82. Jekaterina Novikova, Ondrej Dusek, and Verena Rieser. 2017. The e2e dataset: New challenges for end-to-end generation. ArXiv, abs/1706.09254. · arXiv
  83. Krista Opsahl-Ong, Michael J Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab. 2024. Optimizing instructions and demon- strations for multi-stage language model programs. In Proceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing, page 9340-9366, Miami, Florida, USA. Association for Computational Linguistics. · DOI · Paper page
  84. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi- subject multi-choice dataset for medical domain ques- tion answering. In Conference on health, inference, and learning, pages 248-260. · DOI
  85. PMLR.
  86. Rui Pan, Shuo Xing, Shizhe Diao, Wenhe Sun, Xiang Liu, KaShun Shum, Jipeng Zhang, Renjie Pi, and Tong Zhang. 2024. Plum: Prompt learning using metaheuris- tics. In Findings of the Association for Computational Linguistics: ACL 2024, page 2177-2197, Bangkok, Thailand. Association for Computational Linguistics. · DOI
  87. Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. ArXiv, cs.CL/0409058. · arXiv
  88. Arkil Patel, S. Bhattamishra, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems? In North American Chapter of the Associa- tion for Computational Linguistics.
  89. Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. Wic: the word-in-context dataset for evaluating context-sensitive meaning representations. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1267-1273. · DOI
  90. Silviu Pitis, Michael R Zhang, Andrew Wang, and Jimmy Ba. 2023. Boosted prompt ensembles for large language models. arXiv preprint arXiv:2304.05970. · DOI
  91. Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. Xcopa: A multilingual dataset for causal commonsense reasoning. arXiv preprint arXiv:2005.00333. · arXiv
  92. Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, and Michael Zeng. 2023. Automatic prompt opti- mization with "gradient descent" and beam search. In Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, page 7957-7968, Singapore. Association for Computational Linguistics. · DOI · Paper page
  93. Ye Qi, Devendra Singh Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. 2018. When and why are pre-trained word embeddings useful for neural machine translation? ArXiv, abs/1804.06323. · arXiv
  94. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Conference on Em- pirical Methods in Natural Language Processing.
  95. David Rein, Betty Li Hou, Asa Cooper Stickland, Jack- son Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2023. Gpqa: A graduate-level google-proof q&a benchmark. ArXiv, abs/2311.12022. · arXiv
  96. Melissa Roemmele, Cosmin Adrian Bejan, and An- drew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI spring symposium series.
  97. Subhro Roy and Dan Roth. 2016. Solving general arith- metic word problems. ArXiv, abs/1608.01413. · arXiv
  98. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social iqa: Common- sense reasoning about social interactions. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Process- ing (EMNLP-IJCNLP), pages 4463-4473. · DOI
  99. Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. 2022. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25294. · DOI
  100. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models' sensitivity to spu- rious features in prompt design or: How i learned to start worrying about prompt formatting. In The Twelfth International Conference on Learning Representations.
  101. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Au- tomatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 4222-4235, Online. Association for Computational Linguistics. · DOI · Paper page
  102. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36. · DOI
  103. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2020. Alfworld: Aligning text and em- bodied environments for interactive learning. arXiv preprint arXiv:2010.03768. · DOI
  104. Ankita Sinha, Wendi Cui, Kamalika Das, and Jiaxin Zhang. 2024. Survival of the safest: Towards secure prompt optimization through interleaved multi-objective evolution. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 1016-1027, Miami, Florida, US. Association for Computational Linguistics. · DOI
  105. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for se- mantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical meth- ods in natural language processing, pages 1631-1642.
  106. Alessandro Sordoni, Eric Yuan, Marc-Alexandre Côté, Matheus Pereira, Adam Trischler, Ziang Xiao, Arian Hosseini, Friederike Niedtner, and Nicolas Le Roux. 2023. Joint prompt optimization of stacked llms us- ing variational inference. In Advances in Neural Infor- mation Processing Systems, volume 36, pages 58128- 58151.
  107. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of lan- guage models. arXiv preprint arXiv:2206.04615. · DOI
  108. Hao Sun, Alihan Hüyük, and Mihaela van der Schaar. 2024a. Query-dependent prompt evaluation and opti- mization with offline inverse RL. In The Twelfth Inter- national Conference on Learning Representations.
  109. Hong Sun, Xue Li, Yinchuan Xu, Youkow Homma, Qi Cao, Min Wu, Jian Jiao, and Denis Charles. 2023. Autohint: Automatic prompt optimization with hint gen- eration. arXiv preprint arXiv:2307.07415. · arXiv
  110. Jingwei Sun, Ziyue Xu, Hongxu Yin, Dong Yang, Daguang Xu, Yudong Liu, Zhixu Du, Yiran Chen, and Holger R. Roth. 2024b. Fedbpt: efficient federated black-box prompt tuning for large language models. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org. · DOI
  111. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebas- tian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, et al. 2023. Challenging big-bench tasks and whether chain- of-thought can solve them. In Findings of the Associa- tion for Computational Linguistics: ACL 2023, pages 13003-13051. · DOI
  112. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge. ArXiv, abs/1811.00937. · arXiv
  113. Ellen M Voorhees and Dawn M Tice. 2000. Building a question answering test collection. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pages 200-207. · DOI
  114. Ruochen Wang, Sohyun An, Minhao Cheng, Tianyi Zhou, Sung Ju Hwang, and Cho-Jui Hsieh. 2025. One prompt is not enough: automated construction of a mixture-of-expert prompts. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org. · DOI
  115. Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. 2022a. Scienceworld: Is your agent smarter than a 5th grader? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11279-11298.
  116. William Yang Wang. 2017. "liar, liar pants on fire": A new benchmark dataset for fake news detection. In Annual Meeting of the Association for Computational Linguistics.
  117. Xinyuan Wang, Chenxi Li, Zhen Wang, Fan Bai, Hao- tian Luo, Jiayou Zhang, Nebojsa Jojic, Eric P. Xing, and Zhiting Hu. 2024a. Promptagent: Strategic planning with language models enables expert-level prompt opti- mization. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
  118. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language mod- els with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 13484-13508, Toronto, Canada. Association for Com- putational Linguistics. · DOI · Paper page
  119. Yushi Wang, Jonathan Berant, and Percy Liang. 2015. Building a semantic parser overnight. In Annual Meet- ing of the Association for Computational Linguistics.
  120. Alex Warstadt, Amanpreet Singh, and Samuel R. Bow- man. 2018. Neural network acceptability judgments. Transactions of the Association for Computational Lin- guistics, 7:625-641.
  121. Adina Williams, Nikita Nangia, and Samuel R. Bow- man. 2017. A broad-coverage challenge corpus for sentence understanding through inference. In North American Chapter of the Association for Computational Linguistics.
  122. Yurong Wu, Yan Gao, Bin Benjamin Zhu, Zineng Zhou, Xiaodi Sun, Sheng Yang, Jian-Guang Lou, Zhiming Ding, and Linjun Yang. 2024. StraGo: Harnessing strategic guidance for prompt optimization. In Find- ings of the Association for Computational Linguistics: EMNLP 2024, pages 10043-10061, Miami, Florida, USA. Association for Computational Linguistics. · DOI
  123. Jasper Xian, Saron Samuel, Faraz Khoubsirat, Ronak Pradeep, Md Arafat Sultan, Radu Florian, Salim Roukos, Avirup Sil, Christopher Potts, and Omar Khat- tab. 2024. Prompts as auto-optimized training hyperpa- rameters: Training best-in-class ir models from scratch with 10 gold labels. arXiv preprint arXiv:2406.11706. · DOI
  124. Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Wang Yanggang, Haiyu Li, and Zhilin Yang. 2022. Gps: Ge- netic prompt search for efficient few-shot learning. In Proceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 8162-8171.
  125. Wei Xu, Alan Ritter, William B. Dolan, Ralph Grish- man, and Colin Cherry. 2012. Paraphrasing for style. In International Conference on Computational Linguistics.
  126. Weijia Xu, Andrzej Banburski-Fahey, and Nebojsa Jo- jic. 2024. Reprompting: automated chain-of-thought prompt inference through gibbs sampling. In Proceed- ings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org. · DOI
  127. Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. 2024b. Large language models as optimizers. In The Twelfth International Conference on Learning Representations.
  128. Muchen Yang, Moxin Li, Yongle Li, Zijun Chen, Chong- ming Gao, Junqi Zhang, Yangyang Li, and Fuli Feng. 2024c. Dual-phase accelerated prompt optimization. In Findings of the Association for Computational Lin- guistics: EMNLP 2024, pages 12163-12173, Miami, Florida, USA. Association for Computational Linguis- tics. · DOI
  129. Sheng Yang, Yurong Wu, Yan Gao, Zineng Zhou, Bin Benjamin Zhu, Xiaodi Sun, Jian-Guang Lou, Zhim- ing Ding, Anbang Hu, Yuan Fang, et al. 2024d. Ampo: Automatic multi-branched prompt optimization. arXiv preprint arXiv:2410.08696. · arXiv
  130. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christo- pher D. Manning. 2018. Hotpotqa: A dataset for di- verse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing.
  131. John M. Zelle and Raymond J. Mooney. 1996. Learning to parse database queries using inductive logic program- ming. In AAAI/IAAI, Vol.
  132. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 4791-4800.
  133. Pengwei Zhan, Zhen Xu, Qian Tan, Jie Song, and Ru Xie. 2024. Unveiling the lexical sensitivity of llms: Combinatorial optimization for prompt enhancement. In Conference on Empirical Methods in Natural Lan- guage Processing.
  134. Chenrui Zhang, Lin Liu, Chuyuan Wang, Xiao Sun, Hongyu Wang, Jinpeng Wang, and Mingchen Cai. 2024a. Prefer: prompt ensemble learning via feedback-reflect-refine. In Proceedings of the Thirty- Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence, AAAI'24/IAAI'24/EAAI'24. AAAI Press. · DOI
  135. Lechen Zhang, Tolga Ergen, Lajanugen Logeswaran, Moontae Lee, and David Jurgens. 2024b. Sprig: Im- proving large language model performance by system prompt optimization. ArXiv, abs/2410.14826. · arXiv
  136. Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classifi- cation. In Neural Information Processing Systems.
  137. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuo- han Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA.
  138. Han Zhou, Xingchen Wan, Ivan Vulić, and Anna Ko- rhonen. 2023. Survival of the most influential prompts: Efficient black-box prompt search via clustering and pruning. In Findings of the Association for Computa- tional Linguistics: EMNLP 2023, pages 13064-13077, Singapore. Association for Computational Linguistics. · DOI

Cited by

Sorted by citations

Authors
Ramnath, K., Zhou, K., Guan, S., Mishra, S. S., Qi, X., Shen, Z., Wang, S., Woo, S., Jeoung, S., Wang, Y., Wang, H., Ding, H., Lu, Y., Xu, Z., Zhou, Y., Srinivasan, B., Yan, Q., Chen, Y., Ding, H., Xu, P., & Cheong, L. L.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
Pages
33066-33098
DOI
10.48550/arxiv.2502.16923
arXiv
2502.16923
OpenAlex
W4416035346
Cited by
26
References
0
Citations by year
2026: 25 · 2025: 1
FWCI
24.79
Type
conference paper
Language
en
Open access
Licence
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
prompt optimization, prompt engineering, large language models, prompt tuning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review