CoChat
Sign Up
PreprintCited by 3Topic ModelingNatural Language Processing TechniquesAdversarial Robustness in Machine Learning

What Prompts Don't Say: Understanding and Managing Underspecification in LLM Prompts

Open access
Yang, Chenyang, Shi, Yike, Ma, Qianou, Liu, Michael Xieyang, Kästner, Christian, Wu, Tongshuang·2025·arXiv
DOI 10.48550/arxiv.2505.13360arXiv 2505.13360
Open this paper in CoChat — free accountOpen PDF

Abstract

Prompt underspecification is a common challenge when interacting with LLMs. In this paper, we present an in-depth analysis of this problem, showing that while LLMs can often infer unspecified requirements by default (41.1%), such behavior is fragile: Under-specified prompts are 2x as likely to regress across model or prompt changes, sometimes with accuracy drops exceeding 20%. This instability makes it difficult to reliably build LLM applications. Moreover, simply specifying all requirements does not consistently help, as models have limited instruction-following ability and requirements can conflict. Standard prompt optimizers likewise provide little benefit. To address these issues, we propose requirements-aware prompt optimization mechanisms that improve performance by 4.8% on average over baselines. We further advocate for a systematic process of proactive requirements discovery, evaluation, and monitoring to better manage prompt underspecification in practice.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2505.13360

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A key challenge in prompt-based website generation is that user-provided prompts often lack essential functional or structural requirements of a complete website.

    “A key challenge in prompt-based website generation is that user-provided prompts often lack essential functional or structural requirements of a complete website [75, 78].”

    From A Multi-Agent Framework for Accessible Web Code Generation (2026) · cites this paper as [75]

    The Introduction (page unavailable) defines prompt underspecification as omitting essential requirements, and Section 4.2 (page unavailable) applies requirement-aware optimization to a website-generation task. But the paper does not establish that user-provided website prompts often omit functional or structural requirements needed for a complete website; its website example concerns a front-end development system prompt. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Prompt engineering alone is insufficient to guarantee compliance with formal accessibility standards.

    “This architecture directly addresses concerns raised in prior work that prompt engineering alone is insufficient to guarantee compliance with formal accessibility standards [42, 75]”

    From A Multi-Agent Framework for Accessible Web Code Generation (2026) · cites this paper as [75]

    The Artifact Evolution and Section 5 passages say that ad-hoc prompt iteration can overlook requirements and that managing underspecification requires a more rigorous process. That supports the general concern about prompt engineering alone, but the paper does not address compliance with formal accessibility standards specifically. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Standard LLMs attempt to infer missing requirements when prompts are underspecified, which leads to inconsistent compliance.

    “Standard LLMs attempt to infer missing requirements when prompts are underspecified, which leads to inconsistent compliance [42, 75].”

    From A Multi-Agent Framework for Accessible Web Code Generation (2026) · cites this paper as [75]

  • Claude and GPT-5 generated hundreds of contrast violations, indicating a lack of inherent visual grounding for accessibility-critical design decisions.

    “Claude and GPT-5 generated hundreds of contrast violations, indicating a lack of inherent visual grounding for accessibility-critical design decisions [51, 75].”

    From A Multi-Agent Framework for Accessible Web Code Generation (2026) · cites this paper as [75]

    The paper’s A.3 Process of requirements curation discusses testing GPT-4o and smaller models on prompt requirements; its front-end-development discussion concerns prompt optimization, not visual accessibility testing. Searches for Claude, GPT-5, contrast violations, and visual grounding found no passage reporting the claimed hundreds of violations or drawing that conclusion. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 42

  1. sonnet. Rahul K Arora, Jason Wei, Rebecca Soskin Hicks, Pre- ston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, An- drea Vallone, Alex Beutel, and 1 others. 2025. Healthbench: Evaluating large language models towards improved human health. arXiv preprint arXiv:2505.08775. · arXiv
  2. Hamed Barzamini, Mona Rahimi, Murteza Shahzad, and Hamed Alhoori. 2022. Improving generalizabil- ity of ml-enabled software through domain specifica- tion. In Proceedings of the 1st International Confer- ence on AI Engineering: Software Engineering for AI, CAIN '22, page 181-192, New York, NY, USA. Association for Computing Machinery.
  3. James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for hyper-parameter optimization. Advances in neural information pro- cessing systems, 24.
  4. Chris Bogart, Christian Kästner, James Herbsleb, and Ferdian Thung. 2021. When and how to make break- ing changes: Policies and practices in 18 open source software ecosystems. ACM Transactions on Software Engineering and Methodology (TOSEM), 30(4):1- 56.
  5. Bowen Cao, Deng Cai, Zhisong Zhang, Yuexian Zou, and Wai Lam. 2024. On the worst prompt perfor- mance of large language models. arXiv preprint arXiv:2406.10248. · arXiv
  6. Lingjiao Chen, Matei Zaharia, and James Zou. 2024. How is chatgpt's behavior changing over time? Har- vard Data Science Review, 6(2).
  7. Alexander D'Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, and 1 others. 2022. Underspecification presents challenges for credibility in modern machine learning. Journal of Machine Learning Research, 23(226):1-61.
  8. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing chat language models by scaling high-quality instructional conver- sations. arXiv preprint arXiv:2305.14233. · arXiv
  9. Lisa Dunlap, Krishna Mandal, Trevor Darrell, Jacob Steinhardt, and Joseph E Gonzalez. 2024. Vibecheck: Discover and quantify qualitative differences in large language models. arXiv preprint arXiv:2410.12851. · arXiv
  10. Kunal Handa, Alex Tamkin, Miles McCain, Saffron Huang, Esin Durmus, Sarah Heck, Jared Mueller, Jerry Hong, Stuart Ritchie, Tim Belonax, and 1 oth- ers. 2025. Which economic tasks are performed with ai? evidence from millions of claude conversations. arXiv preprint arXiv:2503.04761. · arXiv
  11. Yining Hong, Christopher S Timperley, and Christian Kästner. 2025. From hazard identification to con- troller design: Proactive and llm-supported safety engineering for ml-powered systems. arXiv preprint arXiv:2502.07974. · arXiv
  12. Chip Huyen. 2022. Designing Machine Learning Sys- tems. O'Reilly Media, USA.
  13. Christian Kästner, Eunsuk Kang, and Sven Apel. 2021. Feature interactions on steroids: On the composition of ml models. arXiv preprint arXiv:2105.06449. · arXiv
  14. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vard- hamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, and 1 others. 2023. Dspy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.03714. · arXiv
  15. Hyuhng Joon Kim, Youna Kim, Cheonbok Park, Jun- yeob Kim, Choonghyun Park, Kang Min Yoo, Sang- goo Lee, and Taeuk Kim. 2024. Aligning language models to explicitly handle ambiguity. arXiv preprint arXiv:2404.11972. · arXiv
  16. Mingqi Li, Karan Aggarwal, Yong Xie, Aitzaz Ahmad, and Stephen Lau. 2024. Learning from contrastive prompts: Automated optimization and adaptation. arXiv preprint arXiv:2409.15199. · arXiv
  17. Jenny T Liang, Melissa Lin, Nikitha Rao, and Brad A Myers. 2024. Prompts are programs too! under- standing how developers build software containing prompts. arXiv preprint arXiv:2409.12447. · arXiv
  18. Wanqin Ma, Chenyang Yang, and Christian Kästner. 2024. (why) is my prompt getting worse? rethink- ing regression testing for evolving llm apis. In Pro- ceedings of the IEEE/ACM 3rd International Confer- ence on AI Engineering-Software Engineering for AI, pages 166-171.
  19. Steve McConnell. 1998. Software project survival guide. Pearson Education.
  20. Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. 2023. Octopack: Instruction tun- ing code large language models. arXiv preprint arXiv:2308.07124. · arXiv
  21. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow in- structions with human feedback. Advances in neural information processing systems, 35:27730-27744.
  22. Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. 2024. Infobench: Evaluating instruction following ability in large lan- guage models. arXiv preprint arXiv:2401.03601. · arXiv
  23. Chandan K. Reddy, Lluís Màrquez, Fran Valero, Nikhil Rao, Hugo Zaragoza, Sambaran Bandyopadhyay, Arnab Biswas, Anlu Xing, and Karthik Subbian. 2022. Shopping queries dataset: A large-scale ESCI benchmark for improving product search. arXiv preprint arXiv:2206.06588. · arXiv
  24. Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, and 1 others. 2021. Multitask prompted training en- ables zero-shot task generalization. arXiv preprint arXiv:2110.08207. · arXiv
  25. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2023. Quantifying language models' sensitiv- ity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. arXiv preprint arXiv:2310.11324. · arXiv
  26. Shreya Shankar, JD Zamfirescu-Pereira, Björn Hart- mann, Aditya Parameswaran, and Ian Arawjo. 2024. Who validates the validators? aligning llm-assisted evaluation of llm outputs with human preferences. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, pages 1-14.
  27. Ian Sommerville. 2015. Software Engineering, 10th edition. Pearson.
  28. Mingjie Sun, Yida Yin, Zhiqiu Xu, J Zico Kolter, and Zhuang Liu. 2025. Idiosyncrasies in large language models. arXiv preprint arXiv:2502.12150. · arXiv
  29. Annalisa Szymanski, Simret Araya Gebreegziabher, Oghenemaro Anuyah, Ronald A Metoyer, and Toby Jia-Jun Li. 2024. Comparing criteria development across domain experts, lay users, and models in large language model evaluation. arXiv preprint arXiv:2410.02054. · arXiv
  30. Mahan Tafreshipour, Aaron Imani, Eric Huang, Eduardo Almeida, Thomas Zimmermann, and Iftekhar Ahmed. 2024. Prompting in the wild: An empirical study of prompt evolution in software repositories. arXiv preprint arXiv:2412.17298. · arXiv
  31. Axel Van Lamsweerde. 2009. Requirements engineer- ing: From system goals to UML models to software, volume 10. Chichester, UK: John Wiley & Sons.
  32. Sanidhya Vijayvargiya, Xuhui Zhou, Akhila Yerukola, Maarten Sap, and Graham Neubig. 2025. Interactive agents to overcome ambiguity in software engineer- ing. arXiv preprint arXiv:2502.13069. · arXiv
  33. Chenyang Yang, Rishabh Rustogi, Rachel Brower- Sinning, Grace Lewis, Christian Kaestner, and Tong- shuang Wu. 2023. Beyond testers' biases: Guiding model testing with knowledge bases using llms. In Findings of the Association for Computational Lin- guistics: EMNLP 2023, pages 13504-13519.
  34. Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. 2024. Textgrad: Automatic" differentiation" via text. arXiv preprint arXiv:2406.07496. · arXiv
  35. J Diego Zamfirescu-Pereira, Richmond Y Wong, Bjoern Hartmann, and Qian Yang. 2023. Why johnny can't prompt: how non-ai experts try (and fail) to design llm prompts. In Proceedings of the 2023 CHI confer- ence on human factors in computing systems, pages 1-21.
  36. Zhiyuan Zeng, Yizhong Wang, Hannaneh Hajishirzi, and Pang Wei Koh. 2025. Evaltree: Profiling lan- guage model weaknesses via hierarchical capability trees. arXiv preprint arXiv:2503.08893. · arXiv
  37. Michael JQ Zhang and Eunsol Choi. 2023. Clarify when necessary: Resolving ambiguity through interaction with lms. arXiv preprint arXiv:2311.09469. · arXiv
  38. Tong Zhang, Peixin Qin, Yang Deng, Chen Huang, Wen- qiang Lei, Junhong Liu, Dingnan Jin, Hongru Liang, and Tat-Seng Chua. 2024. Clamber: A benchmark of identifying and clarifying ambiguous informa- tion needs in large language models. arXiv preprint arXiv:2405.12063. · arXiv
  39. Chenyang Zhao, Xueying Jia, Vijay Viswanathan, Tong- shuang Wu, and Graham Neubig. 2024. Self- guide: Better task-specific instruction follow- ing via self-synthetic finetuning. arXiv preprint arXiv:2407.12874. · arXiv
  40. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Pro- cessing Systems, 36:46595-46623.
  41. Han Zhou, Xingchen Wan, Ruoxi Sun, Hamid Palangi, Shariq Iqbal, Ivan Vulić, Anna Korhonen, and Ser- can Ö Arık. 2025. Multi-agent design: Optimizing agents with better prompts and topologies. arXiv preprint arXiv:2502.02533. · arXiv
  42. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Sid- dhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evalu- ation for large language models. arXiv preprint arXiv:2311.07911. · arXiv

Cited by

Sorted by citations

Authors
Yang, C., Shi, Y., Ma, Q., Liu, M. X., Kästner, C., & Wu, T.
Affiliations
Carnegie Mellon University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
Pages
9072-9101
DOI
10.48550/arxiv.2505.13360
arXiv
2505.13360
OpenAlex
W4417287666
Cited by
3
References
1
Citations by year
2026: 3
FWCI
26.81
Type
conference paper
Language
en
Open access
Licence
Locations
  • Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
large language models, prompt engineering, regression analysis, prompt optimization, instruction following, requirements discovery, LLM applications

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review