Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 112 of the 112 references it cites were found in a registry or an index.

Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant

Open access
He, Gaole, Demartini, Gianluca, Gadiraju, Ujwal·2025·Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems

Abstract

Since the explosion in popularity of ChatGPT, large language models (LLMs) have continued to impact our everyday lives. Equipped with external tools that are designed for a specific purpose (e.g., for flight booking or an alarm clock), LLM agents exercise an increasing capability to assist humans in their daily work. Although LLM agents have shown a promising blueprint as daily assistants, there is a limited understanding of how they can provide daily assistance based on planning and sequential decision making capabilities. We draw inspiration from recent work that has highlighted the value of 'LLM-modulo' setups in conjunction with humans-in-the-loop for planning tasks. We conducted an empirical study (N = 248) of LLM agents as daily assistants in six commonly occurring tasks with different levels of risk typically associated with them (e.g., flight ticket booking and credit card payments). To ensure user agency and control over the LLM agent, we adopted LLM agents in a plan-then-execute manner, wherein the agents conducted step-wise planning and step-by-step execution in a simulation environment. We analyzed how user involvement at each stage affects their trust and collaborative team performance. Our findings demonstrate that LLM agents can be a double-edged sword -- (1) they can work well when a high-quality plan and necessary user involvement in execution are available, and (2) users can easily mistrust the LLM agents with plans that seem plausible. We synthesized key insights for using LLM agents as daily assistants to calibrate user trust and achieve better overall task outcomes. Our work has important implications for the future design of daily assistants and human-AI collaboration with LLM agents.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3706598.3713218

Cites

112 of 112 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 96 references

Research works printed but not linked: 10

  1. Chun-Wei Chiang and Ming Yin. 2022. Exploring the Effects of Machine Learning Literacy Interventions on Laypeople’s Reliance on Machine Learning Models. In IUI 2022: 27th International Conference on Intelligent User Interfaces, Helsinki, Finland, March 22 - 25, 2022, Giulio Jacucci, Samuel Kaski, Cristina Conati, Simone Stumpf, Tuukka Ruotsalo, and Krzysztof Gajos (Eds.). ACM, 148–161.
  2. Patrick Hemmer Max Schemmer Michael Vössing and Niklas Kühl. 2021. Human-AI Complementarity in Hybrid Intelligence Systems: A Structured Literature Review. PACIS (2021) 78.
  3. Patricia K Kahr Gerrit Rooks Martijn C Willemsen and Chris CP Snijders. 2024. Understanding Trust and Reliance Development in AI Advice: Assessing Model Accuracy Model Explanations and Experiences from Previous Interactions. ACM Transactions on Interactive Intelligent Systems (2024).
  4. Zhuoyan Li and Ming Yin. [n. d.]. Utilizing Human Behavior Modeling to Manipulate Explanations in AI-Assisted Decision Making: The Good, the Bad, and the Scary. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
  5. Brian Lubars and Chenhao Tan. 2019. Ask not what AI can do but what AI should do: Towards a framework of task delegability. Advances in neural information processing systems 32 (2019).
  6. David Madras Toni Pitassi and Richard Zemel. 2018. Predict responsibly: improving fairness and accuracy by learning to defer. Advances in neural information processing systems 31 (2018).
  7. Harikrishna Narasimhan Wittawat Jitkrittum Aditya K Menon Ankit Rawat and Sanjiv Kumar. 2022. Post-hoc estimators for learning to defer to an expert. Advances in Neural Information Processing Systems 35 (2022) 29292–29304.
  8. Max Schemmer, Patrick Hemmer, Niklas Kühl, Carina Benz, and Gerhard Satzger. 2022. Should I Follow AI-based Advice? Measuring Appropriate Reliance in Human-AI Decision-Making. In ACM Conference on Human Factors in Computing Systems (CHI’22), Workshop on Trust and Reliance in AI-Human Teams (trAIt).
  9. Zhiping Zhang, Michelle Jia, Hao-Ping Lee, Bingsheng Yao, Sauvik Das, Ada Lerner, Dakuo Wang, and Tianshi Li. 2024. “It’s a Fair Game”, or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1–26.
  10. Yuchen Zhuang Yue Yu Kuan Wang Haotian Sun and Chao Zhang. 2024. Toolqa: A dataset for llm question answering with external tools. Advances in Neural Information Processing Systems 36 (2024).

Cited by

See all 26

Metadata

Authors
He, G., Demartini, G., & Gadiraju, U.
Affiliations
  • Delft University of Technology (NL)
  • The University of Queensland (AU)
Venue
Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2025-04-25
Pages
1-22
OpenAlex
W4407170313
Cited by
23
References
96
Citations by year
2026: 15 · 2025: 8
FWCI
43.92
Type
conference paper
Language
en
Open access
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
Australian Research Council (FT240100022, IC200100022)
Topics
  • Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
Keywords
LLM agents, user trust, team performance, sequential decision making, human-in-the-loop planning, human-AI collaboration, user participation, airline reservations, credit card payments, simulation environment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant. https://app.cochat.ai/papers/10.1145/3706598.3713218
The paper
He, G., Demartini, G., & Gadiraju, U. (2025). Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant. Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3706598.3713218

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.