WildIFEval: Instruction Following in the Wild
Open accessAbstract
Recent LLMs have shown remarkable success in following user instructions, yet handling instructions with multiple constraints remains a significant challenge. In this work, we introduce WildIFEval - a large-scale dataset of 7K real user instructions with diverse, multi-constraint conditions. Unlike prior datasets, our collection spans a broad lexical and topical spectrum of constraints, extracted from natural user instructions. We categorize these constraints into eight high-level classes to capture their distribution and dynamics in real-world scenarios. Leveraging WildIFEval, we conduct extensive experiments to benchmark the instruction-following capabilities of leading LLMs. WildIFEval clearly differentiates between small and large models, and demonstrates that all models have a large room for improvement on such tasks. We analyze the effects of the number and type of constraints on performance, revealing interesting patterns of model constraint-following behavior. We release our dataset to promote further research on instruction-following under complex, realistic conditions.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Users naturally use constraints in their prompts.
“Users naturally use constraints in their prompts [34, 16]”
From Generalizing Verifiable Instruction Following (2025) · cites this paper as [16]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 1× · see citing works
“From Figure 6 we can see that within model families, larger models perform better, in line with previous literature (Kaplan et al., 2020).”
CommonGen: A constrained text generation challenge for generative commonsense reasoning
Lin, B. Y., Zhou, W., Shen, M., Zhou, P., Bhagavatula, C., Choi, Y., & Ren, X. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.165 · Cited by 239 · cited 1× · see citing works
“Recent interest in LLM instruction-following capabilities raises the need for benchmarking model performance under complex, multi-constraint scenarios (Lin et al., 2020; Sun et al., 2023).”
Flagged first, then by citations
Research works printed but not linked: 21
- Mohaddeseh Bastan, Mihai Surdeanu, and Niranjan Balasubramanian. 2023. NEUROSTRUCTURAL training llms as generalist embedding models. arXiv preprint arXiv:2405.17428. · DOI
- Cheng Li, Mingyang Zhang, Qiaozhu Mei, Weize Kong, and Michael Bendersky. 2024a. Learning to rewrite prompts for personalized text generation. In Proceed- ings of the ACM Web Conference 2024, WWW '24, page 3367-3378, New York, NY, USA. Association for Computing Machinery. · DOI
- Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. 2024b. From crowdsourced data to high- quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939. · arXiv
- Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. · arXiv
- Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human align- ment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511-2522, Singapore. Association for Com- putational Linguistics. · DOI · Paper page
- Yotam Perlitz, Ariel Gera, Ofir Arviv, Asaf Yehudai, El- ron Bandel, Eyal Shnarch, Michal Shmueli-Scheuer, and Leshem Choshen. 2024. Do these llm bench- marks agree? fixing benchmark evaluation with benchbench. arXiv preprint arXiv:2407.13696. · arXiv
- Yunjia Qi, Hao Peng, Xiaozhi Wang, Amy Xin, Youfeng Liu, Bin Xu, Lei Hou, and Juanzi Li. 2026. AGEN- TIF: Benchmarking large language models instruc- tion following ability in agentic scenarios. In The Thirty-ninth Annual Conference on Neural Informa- tion Processing Systems Datasets and Benchmarks Track.
- Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. 2024. Infobench: Evaluating instruction following ability in large lan- guage models. arXiv preprint arXiv:2401.03601. · arXiv
- David Rein, Betty Li Hou, Asa Cooper Stickland, Jack- son Petty, Richard Yuanzhe Pang, Julien Dirani, Ju- lian Michael, and Samuel R Bowman. 2023. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022. · arXiv
- Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. 2023. LaMP: When large lan- guage models meet personalization. arXiv preprint arXiv:2304.11406. · arXiv
- Jiao Sun, Yufei Tian, Wangchunshu Zhou, Nan Xu, Qian Hu, Rahul Gupta, John Wieting, Nanyun Peng, and Xuezhe Ma. 2023. Evaluating large language models on controlled generation tasks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3155-3168, Singapore. Association for Computational Linguistics. · DOI · Paper page
- Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupati- raju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, and 1 others. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. · arXiv
- Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research, 9(86):2579-2605.
- Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yix- uan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. 2024. Replacing judges with juries: Evaluating llm genera- tions with a panel of diverse models. arXiv preprint arXiv:2404.18796. · arXiv
- Jiayin Wang, Fengran Mo, Weizhi Ma, Peijie Sun, Min Zhang, and Jian-Yun Nie. 2024. A user-centric multi- intent benchmark for evaluating large language mod- els. In Proceedings of the 2024 Conference on Empir- ical Methods in Natural Language Processing, pages 3588-3612, Miami, Florida, USA. Association for Computational Linguistics. · DOI
- An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Day- iheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. · arXiv
- Shunyu Yao, Howard Chen, Austin W Hanjie, Runzhe Yang, and Karthik Narasimhan. 2023. Collie: Sys- tematic construction of constrained text generation tasks. arXiv preprint arXiv:2307.08689. · arXiv
- Tao Zhang, Yanjun Shen, Wenjing Luo, Yan Zhang, Hao Liang, Fan Yang, Mingan Lin, Yujing Qiao, Weipeng Chen, Bin Cui, and 1 others. 2024. CFbench: A comprehensive constraints-following benchmark for llms. arXiv preprint arXiv:2408.01122. · arXiv
- Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric. P Xing, Joseph E. Gon- zalez, Ion Stoica, and Hao Zhang. 2023a. Lmsys- chat-1m: A large-scale real-world llm conversation dataset. Preprint, arXiv:2309.11998. · arXiv
- Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023b. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Pro- cessing Systems, 36:46595-46623.
- Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Sid- dhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evalu- ation for large language models. arXiv preprint arXiv:2311.07911. "Maintain all facts present in the original sentence.": "Editing", "Do not add new information.": "Include / Avoid", "Use a casual and polite tone.": "Style and Tone" } } · arXiv
Cited by
Sorted by citations
Generalizing Verifiable Instruction Following
Pyatkin, V., Malik, S., Graf, V., Ivison, H., Huang, S., Dasigi, P., Lambert, N., & Hajishirzi, H. (2025) · arXiv · doi:10.48550/arxiv.2507.02833 · Cited by 4 · see citing works
RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Following
Pan, T., Lin, X., Yang, W., He, Q., Chen, S., Qi, L., Xu, W., Feng, H., Xu, B., & Xiao, Y. (2026) · arXiv · doi:10.48550/arxiv.2603.25133
“Representative benchmarks include InfoBench (Qin et al., 2024b), CFBench (Zhang et al., 2025a), TRACE (Zhang et al., 2025b) and Wildifeval(Lior et al., 2025).”
- Authors
- Lior, G., Yehudai, A., Gera, A., & Ein-Dor, L.
- Affiliations
- Hebrew University of Jerusalem (IL)
- IBM (United States) (US)
- ORCID
- Yehudai, Asaf
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- Pages
- 745-778
- arXiv
- 2503.06573
- OpenAlex
- W7166889511
- Cited by
- 6
- References
- 2
- Citations by year
- 2026: 6
- FWCI
- 25.37
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Workshop on Generation, Evaluation and Metrics (GEM) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- instruction following, large language models, benchmarking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
