CoChat
Sign Up
PreprintCited by 2Usability and User Interface DesignArtificial Intelligence ApplicationsNatural Language Processing Techniques

Towards better Human-Agent Alignment: Assessing Task Utility in LLM-Powered Applications

Open access
Arabzadeh, Negar, Kiseleva, Julia, Wu, Qingyun, Wang, Chi, Awadallah, Ahmed, Dibia, Victor, Fourney, Adam, Clarke, Charles·2024·arXiv
DOI 10.48550/arxiv.2402.09015arXiv 2402.09015
Open this paper in CoChat — free accountOpen PDF

Abstract

The rapid development in the field of Large Language Models (LLMs) has led to a surge in applications that facilitate collaboration among multiple agents to assist humans in their daily tasks. However, a significant gap remains in assessing whether LLM-powered applications genuinely enhance user experience and task execution efficiency. This highlights the pressing need for methods to verify utility of LLM-powered applications, particularly by ensuring alignment between the application's functionality and end-user needs. We introduce AgentEval provides an implementation for the math problems, a novel framework designed to simplify the utility verification process by automatically proposing a set of criteria tailored to the unique purpose of any given application. This allows for a comprehensive assessment, quantifying the utility of an application against the suggested criteria. We present a comprehensive analysis of the robustness of quantifier's work.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2402.09015

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Recent studies demonstrate that LLM-based evaluations can correlate with human judgments, in some cases reaching parity with inter-annotator agreement levels.

    “Recent studies demonstrate that LLM-based evaluations can correlate with human judgments, in some cases reaching parity with inter-annotator agreement levels (Arabzadeh et al., 2024).”

    From A Comprehensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systems (2025) · cites this paper as Arabzadeh et al., 2024

    The paper’s ‘Using LLMs as evaluators’ discussion describes LLMs being used alongside human evaluation, and its Conclusions describe them as a possible alternative for open-ended tasks (page numbers unavailable). These passages support the general connection between LLM-based and human evaluation, but the copy does not report a correlation with human judgments or parity with inter-annotator agreement. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • LLM-based evaluation methods are sensitive to prompt design and susceptible to biases introduced by subtle instructional variations.

    “However, these methods are sensitive to prompt design and susceptible to biases introduced by subtle instructional variations (Arabzadeh et al., 2024; Zhao et al., 2025b).”

    From A Comprehensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systems (2025) · cites this paper as Arabzadeh et al., 2024

    Section 6.1 (page unavailable) reports that changing the inputs used to generate evaluation criteria changes the criteria produced, and that the QuantifierAgent's results vary across runs for one criterion. This supports a narrower concern about evaluation stability, but the paper does not establish sensitivity to subtle instructional variations or bias caused by them. (LLM, conf=0.80)

    Experimental — accuracy not yet measured

  • Meta-evaluation approaches position LLM agents as evaluators or safety monitors, offering scalable oversight but exposing limitations of current risk awareness.

    “Meta-evaluation approaches, e.g., Agent-as-a-Judge (Zhuge et al., 2024b), AgentEval (Arabzadeh et al., 2024), and R-Judge (Yuan et al., 2024b) - position LLM agents themselves as evaluators or safety monitors, offering scalable oversight but also exposing the limitations of current 'risk awareness.'”

    From A Comprehensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systems (2025) · cites this paper as Arabzadeh et al., 2024

    The Introduction and Conclusions describe AgentEval as an LLM-agent framework for evaluating task utility at scale, supporting its identification as an evaluator. The paper’s robustness discussion concerns the reliability of evaluation criteria, not safety monitoring or limitations in agents’ “risk awareness.” No page numbers are provided in our copy. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 48

  1. Ali Ahmadvand, Negar Arabzadeh, Julia Kiseleva, Patricio Figueroa Sanz, Xin Deng, Sujay Jauhar, Michael Gamon, Eugene Agichtein, Ned Friend, et al. 2022. Supporting complex information-seeking tasks with implicit constraints. arXiv preprint arXiv:2205.00584. · arXiv
  2. Leif Azzopardi, Paul Thomas, and Nick Craswell. 2018. Measuring the utility of search engine result pages: an information foraging based measure. In The 41st International ACM SIGIR conference on research & development in information retrieval, pages 605- 614.
  3. Muneera Bano, Didar Zowghi, and Jon Whittle. 2023. Exploring qualitative research using llms. arXiv preprint arXiv:2306.13298. · arXiv
  4. Ning Bian, Xianpei Han, Le Sun, Hongyu Lin, Yaojie Lu, and Ben He. 2023. Chatgpt is a knowledgeable but inexperienced solver: An investigation of com- monsense problem in large language models. arXiv preprint arXiv:2303.16421. · arXiv
  5. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2023. A sur- vey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology.
  6. Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evalua- tions? arXiv preprint arXiv:2305.01937. · arXiv
  7. Marc-Alexandre Côté, Akos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, et al. 2019. Textworld: A learning envi- ronment for text-based games. In Computer Games: 7th Workshop, CGW 2018, Held in Conjunction with the 27th International Conference on Artificial In- telligence, IJCAI 2018, Stockholm, Sweden, July 13, 2018, Revised Selected Papers 7, pages 41-75.
  8. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166. · arXiv
  9. Zishan Guo, Renren Jin, Chuang Liu, Yufei Huang, Dan Shi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, Deyi Xiong, et al. 2023. Evaluating large language models: A comprehensive survey. arXiv preprint arXiv:2310.19736. · arXiv
  10. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021a. Aligning ai with shared human values. Pro- ceedings of the International Conference on Learn- ing Representations (ICLR).
  11. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. Measuring mathemati- cal problem solving with the math dataset. arXiv preprint arXiv:2103.03874. · arXiv
  12. Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. 2023. Metagpt: Meta programming for multi-agent collabo- rative framework. arXiv preprint arXiv:2308.00352. · arXiv
  13. Sameer Jain, Vaishakh Keshava, Swarnashree Mysore Sathyendra, Patrick Fernandes, Pengfei Liu, Graham Neubig, and Chunting Zhou. 2023. Multi-dimensional evaluation of text summariza- tion with in-context learning. arXiv preprint arXiv:2306.01200. · arXiv
  14. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146. · arXiv
  15. Julia Kiseleva, Eric Crestan, Riccardo Brigo, and Roland Dittel. 2014. Modelling and detecting changes in user satisfaction. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, pages 1449-1458.
  16. Julia Kiseleva and Maarten de Rijke. 2017. Evaluating personal assistants on mobile devices. arXiv preprint arXiv:1706.04524. · arXiv
  17. Julia Kiseleva, Ziming Li, Mohammad Aliannejadi, Shrestha Mohanty, Maartje ter Hoeve, Mikhail Burtsev, Alexey Skrynnik, Artem Zholus, Alek- sandr Panov, Kavya Srinet, Arthur Szlam, Yuxuan Sun, Katja Hofmann, Marc-Alexandre Côté, Ahmed Awadallah, Linar Abdrazakov, Igor Churin, Putra Manggala, Kata Naszadi, Michiel van der Meer, and Taewoon Kim. 2022a. Interactive grounded language understanding in a collaborative environment: Iglu 2021. In Proceedings of the NeurIPS 2021 Com- petitions and Demonstrations Track, volume 176 of Proceedings of Machine Learning Research, pages 146-161.
  18. PMLR.
  19. Julia Kiseleva, Alexey Skrynnik, Artem Zho- lus, Shrestha Mohanty, Negar Arabzadeh, Marc- Alexandre Côté, Mohammad Aliannejadi, Milagro Teruel, Ziming Li, Mikhail Burtsev, Maartje ter Ho- eve, Zoya Volovikova, Aleksandr Panov, Yuxuan Sun, Kavya Srinet, Arthur Szlam, Ahmed Awadallah, Se- ungeun Rho, Taehwan Kwon, Daniel Wontae Nam, Felipe Bivort Haiek, Edwin Zhang, Linar Abdraza- kov, Guo Qingyam, Jason Zhang, and Zhibin Guo. 2022b. Interactive grounded language understanding in a collaborative environment: Retrospective on iglu 2022 competition. In Proceedings of the NeurIPS 2022 Competitions Track, volume 220 of Proceed- ings of Machine Learning Research, pages 204-216.
  20. PMLR.
  21. Julia Kiseleva, Kyle Williams, Ahmed Hassan Awadal- lah, Aidan C Crook, Imed Zitouni, and Tasos Anas- tasakos. 2016a. Predicting user satisfaction with intelligent assistants. In Proceedings of the 39th In- ternational ACM SIGIR conference on Research and Development in Information Retrieval, pages 45-54.
  22. Julia Kiseleva, Kyle Williams, Jiepu Jiang, Ahmed Has- san Awadallah, Aidan C Crook, Imed Zitouni, and Tasos Anastasakos. 2016b. Understanding user satis- faction with intelligent assistants. In Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval, pages 121-130.
  23. Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023a. Camel: Communicative agents for" mind" exploration of large scale language model society. arXiv preprint arXiv:2303.17760. · arXiv
  24. Qintong Li, Leyang Cui, Lingpeng Kong, and Wei Bi. 2023b. Collaborative evaluation: Exploring the synergy of large language models and humans for open-ended generation evaluation. arXiv preprint arXiv:2310.19740. · arXiv
  25. Ziming Li, Julia Kiseleva, Alekh Agarwal, Maarten de Rijke, and Ryen W White. 2020. Optimizing interactive systems via data-driven objectives. arXiv preprint arXiv:2006.12999. · arXiv
  26. Ziming Li, Dookun Park, Julia Kiseleva, Young-Bum Kim, and Sungjin Lee. 2021. Deus: A data-driven approach to estimate user satisfaction in multi-turn dialogues. arXiv preprint arXiv:2103.01287. · arXiv
  27. Alex Liu and Min Sun. 2023. From voices to valid- ity: Leveraging large language models (llms) for tex- tual analysis of policy stakeholder interviews. arXiv preprint arXiv:2312.01202. · arXiv
  28. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xu- anyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. 2023. Agent- bench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688. · arXiv
  29. Yanchen Liu, Timo Schick, and Hinrich Schütze. 2022. Semantic-oriented unlabeled priming for large-scale language models. arXiv preprint arXiv:2202.06133. · arXiv
  30. Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2023. Gaia: a benchmark for general ai assistants. arXiv preprint arXiv:2311.12983. · arXiv
  31. Vivek Myers, Andre Wang He, Kuan Fang, Homer Rich Walke, Philippe Hansen-Estruch, Ching-An Cheng, Mihai Jalobeanu, Andrey Kolobov, Anca Dragan, and Sergey Levine. 2023. Goal representations for instruction following: A semi-supervised language interface to control. In Proceedings of The 7th Con- ference on Robot Learning, volume 229 of Proceed- ings of Machine Learning Research, pages 3894- 3908. PMLR.
  32. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084. · arXiv
  33. Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019. What makes a good conversation? how controllable attributes affect human judgments. arXiv preprint arXiv:1902.08654. · arXiv
  34. Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2019. ALFRED: A benchmark for interpreting grounded instructions for everyday tasks. CoRR, abs/1912.01734. · arXiv
  35. Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020a. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pages 10740-10749.
  36. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2020b. Alfworld: Aligning text and em- bodied environments for interactive learning. arXiv preprint arXiv:2010.03768. · arXiv
  37. Yashar Talebirad and Amirhossein Nadiri. 2023. Multi- agent collaboration: Harnessing the power of intelli- gent llm agents. arXiv preprint arXiv:2306.03314. · arXiv
  38. Lindia Tjuatja, Valerie Chen, Sherry Tongshuang Wu, Ameet Talwalkar, and Graham Neubig. 2023. Do llms exhibit human-like response biases? a case study in survey design. arXiv preprint arXiv:2311.04076. · arXiv
  39. Teemu Vahtola, Mathias Creutz, and Jörg Tiedemann. 2022. It is not easy to detect paraphrases: Analysing semantic similarity with antonyms and negation us- ing the new semantoneg benchmark. In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 249-262.
  40. Jindong Wang, Xixu HU, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Wei Ye, Haojun Huang, Xiubo Geng, Binxing Jiao, Yue Zhang, and Xing Xie. 2023. On the robustness of chatGPT: An adversarial and out-of-distribution per- spective. In ICLR 2023 Workshop on Trustworthy and Reliable Large-Scale Machine Learning Models.
  41. Kyle Williams, Julia Kiseleva, Aidan C Crook, Imed Zitouni, Ahmed Hassan Awadallah, and Madian Khabsa. 2016a. Detecting good abandonment in mo- bile search. In Proceedings of the 25th International Conference on World Wide Web, pages 495-505.
  42. Kyle Williams, Julia Kiseleva, Aidan C Crook, Imed Zitouni, Ahmed Hassan Awadallah, and Madian Khabsa. 2016b. Is this your final answer? evalu- ating the effect of answers on good abandonment in mobile search. In Proceedings of the 39th Inter- national ACM SIGIR conference on Research and Development in Information Retrieval, pages 889- 892.
  43. Kyle Williams and Imed Zitouni. 2017. Does that mean you're happy? rnn-based modeling of user inter- action sequences to detect good abandonment. In Proceedings of the 2017 ACM on Conference on In- formation and Knowledge Management, pages 727- 736.
  44. Terry Winograd. 1972. Understanding natural language. Cognitive psychology, 3(1):1-191.
  45. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. 2023. Auto- gen: Enabling next-gen llm applications via multi- agent conversation framework. arXiv preprint arXiv:2308.08155. · arXiv
  46. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629. · arXiv
  47. Chi Zhang, Penglin Cai, Yuhui Fu, Haoqi Yuan, and Zongqing Lu. 2023. Creative agents: Empowering agents with imagination for creative tasks. arXiv preprint arXiv:2312.02519. · arXiv
  48. Zhuang Ziyu, Chen Qiguang, Ma Longxuan, Li Mingda, Han Yi, Qian Yushan, Bai Haopeng, Zhang Weinan, and Ting Liu. 2023. Through the lens of core compe- tency: Survey on evaluation of large language mod- els. In Proceedings of the 22nd Chinese National Conference on Computational Linguistics (Volume 2: Frontier Forum), pages 88-109, Harbin, China. Chinese Information Processing Society of China.

Cited by

Sorted by citations

Authors
Arabzadeh, N., Kiseleva, J., Wu, Q., Wang, C., Awadallah, A., Dibia, V., Fourney, A., & Clarke, C.
Affiliations
  • University of Waterloo (CA)
  • Microsoft (United States) (US)
  • Pennsylvania State University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2402.09015
arXiv
2402.09015
OpenAlex
W4391835799
Cited by
1
References
0
Citations by year
2024: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
agent cooperation, mathematical problem solving, user experience

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review