The DOI matches its Crossref record; 112 of the 112 references it cites were found in a registry or an index.
Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant
Open accessAbstract
Since the explosion in popularity of ChatGPT, large language models (LLMs) have continued to impact our everyday lives. Equipped with external tools that are designed for a specific purpose (e.g., for flight booking or an alarm clock), LLM agents exercise an increasing capability to assist humans in their daily work. Although LLM agents have shown a promising blueprint as daily assistants, there is a limited understanding of how they can provide daily assistance based on planning and sequential decision making capabilities. We draw inspiration from recent work that has highlighted the value of 'LLM-modulo' setups in conjunction with humans-in-the-loop for planning tasks. We conducted an empirical study (N = 248) of LLM agents as daily assistants in six commonly occurring tasks with different levels of risk typically associated with them (e.g., flight ticket booking and credit card payments). To ensure user agency and control over the LLM agent, we adopted LLM agents in a plan-then-execute manner, wherein the agents conducted step-wise planning and step-by-step execution in a simulation environment. We analyzed how user involvement at each stage affects their trust and collaborative team performance. Our findings demonstrate that LLM agents can be a double-edged sword -- (1) they can work well when a high-quality plan and necessary user involvement in execution are available, and (2) users can easily mistrust the LLM agents with plans that seem plausible. We synthesized key insights for using LLM agents as daily assistants to calibrate user trust and achieve better overall task outcomes. Our work has important implications for the future design of daily assistants and human-AI collaboration with LLM agents.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Most AI Agents require explicit user instruction to act and lack the capacity to formulate or re-prioritize goals dynamically based on changing context/environment or evolving objectives [197].”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [197]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Plausible plans can induce inappropriate reliance [61].”
From Caring Without Feeling: Affective Dynamics as the Control Layer of Human-AI Agent Collaboration (2026) · cites this paper as [61]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Proactivity is similarly underdeveloped.”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [197]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical power analyses using G∗Power 3.1: tests for correlation and regression analyses
Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009) · Behavior Research Methods · doi:10.3758/brm.41.4.1149 · Cited by 37,615
The magical number seven, plus or minus two: Some limits on our capacity for processing information.
Miller, G. A. (1956) · Psychological Review · doi:10.1037/h0043158 · Cited by 17,812
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605
Loss aversion in riskless choice: A reference-dependent model.
Tversky, A., & Kahneman, D. (1991) · The Quarterly Journal of Economics · doi:10.2307/2937956 · Cited by 6,432
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
Algorithm aversion: People erroneously avoid algorithms after seeing them err.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000033 · Cited by 3,232 · cited 2×
“However, on tasks entailing potential risks ( e.g., monetary payments or hiring an employee), humans hesitate to trust such AI systems due to loss aversion [96] and algorithmic aversion [13, 19, 40, 71].”
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Proceedings of the 13th International Conference on Neural Information Processing Systems
Leen, T. K., Dietterich, T. G., & Tresp, V. (2000) · Cited by 2,190
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187 · cited 1×
“With this in the backdrop, existing work on automated task completion has revealed that LLM agents can exhibit promising performance in handling complex tasks like playing games [105], answering complex questions [116], and in simulating social behavior [78].”
Research works printed but not linked: 10
- Chun-Wei Chiang and Ming Yin. 2022. Exploring the Effects of Machine Learning Literacy Interventions on Laypeople’s Reliance on Machine Learning Models. In IUI 2022: 27th International Conference on Intelligent User Interfaces, Helsinki, Finland, March 22 - 25, 2022, Giulio Jacucci, Samuel Kaski, Cristina Conati, Simone Stumpf, Tuukka Ruotsalo, and Krzysztof Gajos (Eds.). ACM, 148–161.
- Patrick Hemmer Max Schemmer Michael Vössing and Niklas Kühl. 2021. Human-AI Complementarity in Hybrid Intelligence Systems: A Structured Literature Review. PACIS (2021) 78.
- Patricia K Kahr Gerrit Rooks Martijn C Willemsen and Chris CP Snijders. 2024. Understanding Trust and Reliance Development in AI Advice: Assessing Model Accuracy Model Explanations and Experiences from Previous Interactions. ACM Transactions on Interactive Intelligent Systems (2024).
- Zhuoyan Li and Ming Yin. [n. d.]. Utilizing Human Behavior Modeling to Manipulate Explanations in AI-Assisted Decision Making: The Good, the Bad, and the Scary. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
- Brian Lubars and Chenhao Tan. 2019. Ask not what AI can do but what AI should do: Towards a framework of task delegability. Advances in neural information processing systems 32 (2019).
- David Madras Toni Pitassi and Richard Zemel. 2018. Predict responsibly: improving fairness and accuracy by learning to defer. Advances in neural information processing systems 31 (2018).
- Harikrishna Narasimhan Wittawat Jitkrittum Aditya K Menon Ankit Rawat and Sanjiv Kumar. 2022. Post-hoc estimators for learning to defer to an expert. Advances in Neural Information Processing Systems 35 (2022) 29292–29304.
- Max Schemmer, Patrick Hemmer, Niklas Kühl, Carina Benz, and Gerhard Satzger. 2022. Should I Follow AI-based Advice? Measuring Appropriate Reliance in Human-AI Decision-Making. In ACM Conference on Human Factors in Computing Systems (CHI’22), Workshop on Trust and Reliance in AI-Human Teams (trAIt).
- Zhiping Zhang, Michelle Jia, Hao-Ping Lee, Bingsheng Yao, Sauvik Das, Ada Lerner, Dakuo Wang, and Tianshi Li. 2024. “It’s a Fair Game”, or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1–26.
- Yuchen Zhuang Yue Yu Kuan Wang Haotian Sun and Chao Zhang. 2024. Toolqa: A dataset for llm question answering with external tools. Advances in Neural Information Processing Systems 36 (2024).
Cited by
AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103599 · Cited by 671
“Most AI Agents require explicit user instruction to act and lack the capacity to formulate or re-prioritize goals dynamically based on changing context/environment or evolving objectives [197].”
AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i3.15161 · Cited by 186
Zheng, W., & Arakawa, Y. (2025) · IEEE Access · doi:10.1109/access.2025.3598088 · Cited by 11
Dong, A., Li, P., Chen, Y., Gibson, S., Zhao, L., & He, M. (2026) · Sustainability · doi:10.3390/su18115313 · Cited by 5
An LLM-Based Agentic Natural Language Dialogue System Approach for Helicopter MUM-T Operations
Künzel, D., Jamakatel, P., Shillig, T., & Schulte, A. (2025) · IEEE International Conference on Human-Machine Systems (ICHMS) · doi:10.1109/ichms65439.2025.11154241 · Cited by 4
Metadata
- Authors
- He, G., Demartini, G., & Gadiraju, U.
- Affiliations
- Delft University of Technology (NL)
- The University of Queensland (AU)
- Venue
- Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2025-04-25
- Pages
- 1-22
- OpenAlex
- W4407170313
- Cited by
- 23
- References
- 96
- Citations by year
- 2026: 15 · 2025: 8
- FWCI
- 43.92
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Australian Research Council (FT240100022, IC200100022)
- Topics
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Keywords
- LLM agents, user trust, team performance, sequential decision making, human-in-the-loop planning, human-AI collaboration, user participation, airline reservations, credit card payments, simulation environment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant. https://app.cochat.ai/papers/10.1145/3706598.3713218
- The paper
- He, G., Demartini, G., & Gadiraju, U. (2025). Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant. Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3706598.3713218
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.