InFoBench: Evaluating Instruction Following Ability in Large Language Models
Open accessAbstract
This paper introduces the Decomposed Requirements Following Ratio (DRFR), a new metric for evaluating Large Language Models' (LLMs) ability to follow instructions.Addressing a gap in current methodologies, DRFR breaks down complex instructions into simpler criteria, facilitating a detailed analysis of LLMs' compliance with various aspects of tasks.Alongside this metric, we present INFOBENCH, a benchmark comprising 500 diverse instructions and 2,250 decomposed questions across multiple constraint categories.Our experiments compare DRFR with traditional scoring methods and explore annotation sources, including human experts, crowdsourced workers, and GPT-4.The findings demonstrate DRFR's higher reliability and the effectiveness of using GPT-4 as a costefficient annotator.The evaluation of several advanced LLMs using this framework reveals their strengths and areas needing improvement, particularly in complex instructionfollowing.This study contributes a novel metric and benchmark, offering insights for future LLM development and evaluation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
InFoBench evaluates models by decomposing the instruction into atomic constraints and rating each constraint individually with an LLM-as-a-judge.
“InFoBench [19] evaluates models by decomposing the instruction into atomic constraints and rating each constraint individually with an LLM-as-a-judge.”
From Generalizing Verifiable Instruction Following (2025) · cites this paper as [19]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · see citing works
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · see citing works
Self-citationScaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441 · see citing works
Self-citationBang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.45 · Cited by 682 · see citing works
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · see citing works
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2018) · arXiv · doi:10.48550/arxiv.1804.07461 · Cited by 562 · see citing works
Is ChatGPT a General-Purpose Natural Language Processing Task Solver?
Qin, C., Zhang, A., Zhang, Z., Chen, J., Yasunaga, M., & Yang, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.85 · Cited by 519 · see citing works
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.05862 · Cited by 442 · see citing works
Measuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291 · see citing works
Holistic Evaluation of Language Models
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.09110 · Cited by 255 · see citing works
A General Language Assistant as a Laboratory for Alignment
Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.00861 · Cited by 146 · see citing works
Improving alignment of dialogue agents via targeted human judgements
Glaese, A., McAleese, N., Trębacz, M., Aslanides, J., Firoiu, V., Ewalds, T., Rauh, M., Weidinger, L., Chadwick, M., Thacker, P., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.14375 · Cited by 134 · see citing works
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization
Iyer, S., Lin, X. V., Pasunuru, R., Mihaylov, T., Simig, D., Yu, P., Shuster, K., Wang, T., Liu, Q., Koura, P. S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.12017 · Cited by 81 · see citing works
Large Language Model Instruction Following: A Survey of Progresses and Challenges
Lou, R., Zhang, K., & Yin, W. (2023) · Computational Linguistics · doi:10.1162/coli_a_00523 · Cited by 74 · see citing works
Kenton, Z., Everitt, T., Weidinger, L., Gabriel, I., Mikulik, V., & Irving, G. (2021) · arXiv · doi:10.48550/arxiv.2103.14659 · Cited by 45 · see citing works
InstructEval: Towards Holistic Evaluation of Instruction-Tuned Large Language Models
Chia, Y. K., Hong, P., Bing, L., & Poria, S. (2024) · doi:10.18653/v1/2024.scalellm-1.4 · Cited by 36 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
- Paul F Christiano, Jan Leike, Tom Brown, Miljan Mar- tic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Ad- vances in neural information processing systems, 30.
- Joseph L Fleiss. 1971. Measuring nominal scale agree- ment among many raters. Psychological bulletin, 76(5):378.
- Renze Lou, Kai Zhang, and Wenpeng Yin. 2023. Is prompt all you need? no. a comprehensive and broader view of instruction learning. arXiv preprint arXiv:2303.10475. · arXiv
- Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stick- ier benchmark for general-purpose language under- standing systems. Advances in neural information processing systems, 32.
Cited by
Sorted by citations
From Matching to Generation: A Survey on Generative Information Retrieval
Li, X., Jin, J., Zhou, Y., Zhang, Y., Zhang, P., Zhu, Y., & Dou, Z. (2025) · ACM Transactions on Information Systems · doi:10.1145/3722552 · Cited by 124 · see citing works
Exploring the Impact of Temperature on Large Language Models: Hot or Cold?
Li, L., Sleem, L., Gentile, N., Nichil, G., & State, R. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.07.135 · Cited by 62 · see citing works
Tam, Z. R., Wu, C.-K., Tsai, Y.-L., Lin, C.-Y., Lee, H.-Y., & Chen, Y.-N. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-industry.91 · Cited by 62 · see citing works
What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use
Ma, Q., Peng, W., Yang, C., Shen, H., Koedinger, K. R., & Wu, T. (2025) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3731756 · Cited by 51 · see citing works
Sammour, F., Xu, J., Wang, X., Hu, M., & Zhang, Z. (2025) · Journal of Construction Engineering and Management · doi:10.1061/jcemd4.coeng-16639 · Cited by 26 · see citing works
Self-citationHe, Q., Zeng, J., He, Q., Liang, J., & Xiao, Y. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.637 · Cited by 23 · see citing works
CoPrompter: User-Centric Evaluation of LLM Instruction Alignment for Improved Prompt Engineering
Joshi, I., Shahid, S., Venneti, S., Vasu, M., Zheng, Y., Li, Y., Krishnamurthy, B., & Chan, G. Y.-Y. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712102 · Cited by 20 · see citing works
Direct Judgement Preference Optimization
Wang, P., Xu, A., Zhou, Y., Xiong, C., & Joty, S. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.103 · Cited by 16 · see citing works
Toward Verifiable Instruction-Following Alignment for Retrieval Augmented Generation
Dong, G., Song, X., Zhu, Y., Qiao, R., Dou, Z., & Wen, J.-R. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i22.34551 · Cited by 14 · see citing works
The Price of Format: Diversity Collapse in LLMs
Yun, L., An, C., Wang, Z., Peng, L., & Shang, J. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.836 · Cited by 9 · see citing works
MedReadCtrl: Personalizing medical text generation with readability-controlled instruction learningPreprint
Tran, H., Yao, Z., Jang, W. S., Sultana, S., Chang, A., Zhang, Y., & Hong, Y. (2025) · medRxiv · doi:10.1101/2025.07.09.25331239 · Cited by 9 · see citing works
LLMs can be easily Confused by Instructional Distractions
Hwang, Y., Kim, Y., Koo, J., Kang, T., Bae, H., & Jung, K. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.957 · Cited by 8 · see citing works
Beyond Instruction Following: Evaluating Inferential Rule Following of Large Language Models
Sun, W., Zhang, C., Zhang, X., Yu, X., Huang, Z., Xu, H., He, S., Zhao, J., & Liu, K. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-2725-0_25 · Cited by 8 · see citing works
Constraint Back-translation Improves Complex Instruction Following of Large Language Models
Qi, Y., Peng, H. Y., Wang, X., Xu, B., Hou, L., & Li, J. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761324 · Cited by 8 · see citing works
Self-citation(2025) · doi:10.18653/v1/2025.naacl-long.303 · Cited by 7 · see citing works
ReadCtrl: Personalizing text generation with readability-controlled instruction learning
Tran, H., Yao, Z., Li, L., & Yu, H. (2025) · Workshop on Intelligent and Interactive Writing Assistants (In2Writing) · doi:10.18653/v1/2025.in2writing-1.3 · Cited by 7 · see citing works
Step-by-Step Mastery: Enhancing Soft Constraint Following Ability of Large Language Models
Ren, Q., Zeng, J., He, Q., Liang, J., Xiao, Y., Zhou, W., Sun, Z., & Yu, F. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.1004 · Cited by 7 · see citing works
WildIFEval: Instruction Following in the Wild
Lior, G., Yehudai, A., Gera, A., & Ein-Dor, L. (2026) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2026.gem-main.66 · Cited by 6 · see citing works
Yifei, L. S., Chang, A., Malaviya, C., & Yatskar, M. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.732 · Cited by 6 · see citing works
Beyond Facts: Evaluating Intent Hallucination in Large Language Models
Hao, Y., Yu, H., & You, J. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.349 · Cited by 6 · see citing works
UltraIF: Advancing Instruction Following from the Wild
An, K., Sheng, L., Cui, G., Si, S., Ding, N., Cheng, Y., & Chang, B. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.945 · Cited by 6 · see citing works
Jin, Z., Liu, S., Li, H., Zhao, X., & Qu, H. (2025) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2025.3557568 · Cited by 6 · see citing works
TAKAGI, H., Moriya, S., Sato, T., Nagao, M., & Higuchi, K. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712119 · Cited by 5 · see citing works
AgentSense: LLMs Empower Generalizable and Explainable Web-Based Participatory Urban Sensing
Guo, X., Peng, M., Hao, X., Zou, X., Wang, Q., Ruan, S., & Liang, Y. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792406 · Cited by 5 · see citing works
FLEX: A Benchmark for Evaluating Robustness of Fairness in Large Language Models
Jung, D., Lee, S., Moon, H., Park, C., & Lim, H. (2025) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2025.findings-naacl.199 · Cited by 5 · see citing works
Generalizing Verifiable Instruction Following
Pyatkin, V., Malik, S., Graf, V., Ivison, H., Huang, S., Dasigi, P., Lambert, N., & Hajishirzi, H. (2025) · arXiv · doi:10.48550/arxiv.2507.02833 · Cited by 4 · see citing works
StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction Following
Li, J., Li, J., Wang, Y., Chang, Y., & Wu, Y. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.486 · Cited by 4 · see citing works
Wu, X., Wang, M., Liu, Y., Shi, X., Yan, H., Xiangju, L., Zhu, J., & Zhang, W. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.803 · Cited by 4 · see citing works
Self-citationHypoEval: Hypothesis-Guided Evaluation for Natural Language Generation
Li, M., Li, H., & Tan, C. (2026) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2026.acl-long.1963 · Cited by 4 · see citing works
Yan, K., Guo, H., Shi, X., Cao, S., Di, D., & Li, Z. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-industry.89 · Cited by 4 · see citing works
MultiConIR: Towards Multi-Condition Information Retrieval
Lu, X., Liu, S., Yin, B., Li, Y., Chen, X., Su, H., Jin, Y., Zeng, W., & Shen, X. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.726 · Cited by 3 · see citing works
PROPEL: Prompt Optimization with Expert Priors for Small and Medium-sized LLMs
Mayilvaghanan, K., Nathan, V., & Kumar, A. (2025) · International Workshop on Knowledge-Augmented Methods for Natural Language Processing (KnowledgeNLP) · doi:10.18653/v1/2025.knowledgenlp-1.25 · Cited by 3 · see citing works
LMUNIT: Fine-grained Evaluation with Natural Language Unit Tests
Saad-Falcon, J., Vivek, R., Berrios, W., Naik, N. S., Franklin, M., Vidgen, B., Singh, A., Kiela, D., & Mehri, S. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.176 · Cited by 2 · see citing works
Enhancing and Assessing Instruction-Following with Fine-Grained Instruction Variants
Yang, J., Liu, H., Guo, W., Yu, X., & Niu, D. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761322 · Cited by 2 · see citing works
Vir, R., Shankar, S., Chase, H., Hinthorn, W., & Parameswaran, A. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.213 · Cited by 1 · see citing works
Be Cautious When Merging Unfamiliar LLMs: A Phishing Model Capable of Stealing Privacy
Zhenyuan, G., Shi, Y., Meng, W., Gong, C., Wei, C., & Chen, W. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.713 · Cited by 1 · see citing works
DICE: Structured Reasoning in LLMs through SLM-Guided Chain-of-Thought Correction
Li, Y., Liao, Y., Chen, Z., Wang, Y., & Wang, Y. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.355 · Cited by 1 · see citing works
Learning to Follow Domain-specific Instruction with Verifiable Rewards
Huang, S., Liu, Y., Qi, J., Yang, H., Luan, Z., & Qian, D. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228901 · Cited by 1 · see citing works
Two Shots are Enough: Reliable Constrained Generation with LLMs
Mondal, M., Dolamic, L., Cudré-Mauroux, P., & Audiffren, J. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402199 · Cited by 1 · see citing works
ISA-Bench: Benchmarking Instruction Sensitivity for Large Audio Language Models
Li, B. R., Huang, W., Qiu, Y., Guo, Y., Wang, H., Li, Z., Peng, J., Ma, Z., Chen, X., & Yu, K. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11462799 · Cited by 1 · see citing works
OctoBench: Benchmarking Scaffold-Aware Instruction Following in Repository-Grounded Agentic Coding
Ding, D., Liu, S., Yang, E., Lin, J., Chen, Z., Dou, S., Guo, H., Cheng, W., Zhao, P., Xiao, C., et al. (2026) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2026.acl-long.269 · Cited by 1 · see citing works
Empowering Reliable Visual-Centric Instruction Following in MLLMs
He, W., Ju, F., Fan, Z., Min, R., & Cheng, M. (2026) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2026.findings-acl.461 · Cited by 1 · see citing works
Guerrero, C. U., Corcho, Ó., & Garijo, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-65794-8_8 · Cited by 1 · see citing works
TOWER: Tree Organized Weighting for Evaluating Complex Instructions
Ziems, N., Zhang, Z., & Jiang, M. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.809 · Cited by 1 · see citing works
Evolutionary Contrastive Distillation for Language Model Alignment
Katz-Samuels, J., Li, Z., Yun, H., Nigam, P., Xu, Y., Petricek, V., Yin, B., & Chilimbi, T. M. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.307 · Cited by 1 · see citing works
Chasing Random: Instruction Selection Strategies Fail to Generalize
Diddee, H., & Ippolito, D. (2025) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2025.findings-naacl.103 · Cited by 1 · see citing works
Order Matters: Investigate the Position Bias in Multi-constraint Instruction Following
Zeng, J., He, Q., Ren, Q., Liang, J., Zhou, W., Sun, Z., Yu, F., & Xiao, Y. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.646 · Cited by 1 · see citing works
Chatbot Construction for Business-Oriented Use CasesPreprint
Kováč, J., Viktorin, A., & Šenkeřík, R. (2024) · Research Square · doi:10.21203/rs.3.rs-4278245/v1
Fan, Z., He, J., Li, X., Kuang, S., Song, K., Zhou, Y., & Qiu, X. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.438
Self-citationAligning to Constraints for Data-Efficient Language Model Customization
Fei, W., Shang, C., Wang, S., Jain, S., Ning, Q., Min, B., Castelli, V., Benajiba, Y., & Roth, D. (2025) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2025.findings-naacl.294
Adaptive Contrastive Decoding for Instruction-Following in Large Language Models
Zhang, J., Du, J., & Xue, Z. (2025) · International Conference on Artificial Intelligence and Big Data (ICAIBD) · doi:10.1109/icaibd64986.2025.11082001
Reverse Preference Optimization for Complex Instruction Following
Huang, X. H., Lin, T.-E., Fang, F., Wu, Y., Li, H., Qu, Y., Huang, F., & Li, Y. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.1064
ULTRABENCH: Benchmarking LLMs under Extreme Fine-grained Text Generation
Yun, L., Peng, L., & Shang, J. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.835
When Truthful Representations Flip Under Deceptive Instructions?
Long, X., Fu, Y., Li, R., Sheng, M., Yu, H., Han, X., & Li, P. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.826
REFER: Mitigating Bias in Opinion Summarisation via Frequency Framed Prompting
Huang, N., Fayek, H. M., & Zhang, X. (2025) · Workshop on New Frontiers in Summarization (NewSum) · doi:10.18653/v1/2025.newsum-main.6
From Intentions to Actions: Workflow Generation with LLMs for Robots
Flerlage, J., Acker, A., Scheinert, D., Becker, S., Yous, H., Reddy, K., Farhat, I., Hacid, H., & Kao, O. (2025) · International Conference on Foundation and Large Language Models (FLLM) · doi:10.1109/fllm67465.2025.11391122
A Benchmarkable Modular Tool-Augmented Chatbot Architecture for Business Use Cases
Kováč, J., Šenkeřík, R., Viktorin, A., & Kadavý, T. (2026) · IEEE Access · doi:10.1109/access.2026.3672311
Scaling Towards the Information Boundary of Instructions through Data Synthesizing
Du, L., Zhao, H., Ju, Y., & Pan, T. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i36.40311
Wang, C., Wen, L., Jia, S., Zhang, X. B., & Xu, L. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i39.40629
We Are What We Repeatedly Do: Improving Long Context Instruction Following
Robinette, P. K., Hard, A., Ramaswamy, S., Amid, E., Mathews, R., & Johnson, T. T. (2026) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2026.findings-eacl.254
DIALEVAL: Automated Type-Theoretic Evaluation of LLM Instruction Following
Basta, N., & Kaafar, D. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-1468-6_13
Embedding AI agents into computational design workflows with physics simulation tools
Lindsay, M. B., Maschmann, M. R., Fernandez, S., Mick, E. J., Mischkot, M., Anderson, D. T., Kovaleski, S. D., & Malof, J. M. (2026) · Advanced Optics for Imaging Applications: UV through LWIR · doi:10.1117/12.3094699
Zheng, Z., Qiu, T., Zhao, Y., & Chen, S. (2026) · IEEE Pacific Visualization Conference (PacificVis) · doi:10.1109/pacificvis68791.2026.00042
PALM: Scaling Physiologically-Aware AI Tutoring Through Consumer Wearables and Large Language Models
Chang, C.-K., & Li, K.-H. (2026) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3774398.3811576
DAC-Bench: A Decision-Aware Benchmark for Compositional Mobile GUI Tasks
Zhang, Y., Sheng, H., Hu, X., Zhang, S., & Wu, F. (2026) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2026.acl-long.2064
Li, Y., Wang, Z., Fu, T., Cui, G., Yang, S., & Cheng, Y. (2026) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2026.findings-acl.1568
Cheng, Y., Moakhar, A. S., Fan, C., Hosseini, P., Faghih, K., Sodagar, Z., Wang, W., & Feizi, S. (2026) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2026.findings-acl.1848
Mu, R., Wang, Z., Wang, Q., Ma, M., Wang, Z., Liu, M., & Qin, B. (2026) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2026.acl-long.1136
Wang, J., Zhao, Y., Ding, P., Kuang, J., Shen, Y., Zhe, T., Jin, Y., Wang, Z., Li, X., & Cao, X. (2026) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2026.findings-acl.725
Complex-IF and Beyond: Expert Rubrics for RLVR
Mehta, S., Panavas, L., Fleming, E., Mains, P., & Chen, E. (2026) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2026.gem-main.61
Shuttle Between Symbolic Instructions and Neural Parameters of Large Language Models
Sun, W., Xu, H., Liao, H., Yu, X., Jiang, Z., He, S., Zhao, J., & Liu, K. (2026) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2026.acl-long.1269
Bridging Auxiliary Constraints to Resolve Instruction Following in Large Reasoning Models
Zhao, Z., Zhang, S., Wang, H., Wang, Z., Zhao, Y., Zheng, Y., Li, B., He, Y., Wong, K., & Wu, X. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.730
Self-citationInstruction fragility under hidden operational requirementsPreprint
Park, C., Gong, J., Kim, M. Y., Gong, G., Lee, W., Kwak, Y., & Choi, S. (2026) · Research Square · doi:10.21203/rs.3.rs-10014579/v1
EVADE-Bench: Multimodal Benchmark for Evaluating and Enhancing Evasive Content Detection
Xu, A., Yang, Z., Li, J., Yuan, G., Chen, L., Yan, L., Zhou, J., Qin, Z., Chang, H., Chen, Y., et al. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808579
Hoang, T. D., Huynh, T. T., Weidlich, M., Nguyên, T. T., Chen, T., Yin, H., & Nguyen, Q. V. H. (2026) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde65706.2026.00183
A survey on AI agent security: Reasoning, Acting, and Self-Evolving
Liu, D., Luo, T., Huang, S., Yang, J., Liang, S., Jia, X., Lau, C. P., Dong, J., Liu, Y., Bai, Y., et al. (2026) · AI Plus · doi:10.59717/ipj.aiplus.2026.100004
Prosa: Rubric-Based Evaluation of LLMs on Real User Chats in Brazilian Portuguese
Malaquias, R. D., Bonás, G. K., Almeida, T. S., Abonizio, H., Laitz, T., Pires, R., Piau, M., Larcher, C. H. N., & Nogueira, R. F. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39904-5_8
Kumar, A. (2026) · American Journal of Technology · doi:10.58425/ajt.v5i12.624
- Authors
- Qin, Y., Song, K., Hu, Y., Yao, W., Cho, S., Wang, X., Wu, X., Liu, F., Liu, P., & Yu, D.
- Affiliations
- University of Central Florida (US)
- University of Georgia (US)
- Emory University (US)
- Shanghai Jiao Tong University (CN)
- Venue
- Findings of the Association for Computational Linguistics ACL 2024
- Published
- 2024
- Pages
- 13025-13048
- OpenAlex
- W4402684226
- Cited by
- 77
- References
- 17
- Citations by year
- 2026: 29 · 2025: 42 · 2024: 6
- FWCI
- 18.61
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- instruction following, large language models, evaluation metrics, benchmark, annotation reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
