Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant
Open accessThe DOI matches its Crossref record; 114 of the 114 references it cites were found in a registry or an index.
Abstract
Since the explosion in popularity of ChatGPT, large language models (LLMs) have continued to impact our everyday lives. Equipped with external tools that are designed for a specific purpose (e.g., for flight booking or an alarm clock), LLM agents exercise an increasing capability to assist humans in their daily work. Although LLM agents have shown a promising blueprint as daily assistants, there is a limited understanding of how they can provide daily assistance based on planning and sequential decision making capabilities. We draw inspiration from recent work that has highlighted the value of 'LLM-modulo' setups in conjunction with humans-in-the-loop for planning tasks. We conducted an empirical study (N = 248) of LLM agents as daily assistants in six commonly occurring tasks with different levels of risk typically associated with them (e.g., flight ticket booking and credit card payments). To ensure user agency and control over the LLM agent, we adopted LLM agents in a plan-then-execute manner, wherein the agents conducted step-wise planning and step-by-step execution in a simulation environment. We analyzed how user involvement at each stage affects their trust and collaborative team performance. Our findings demonstrate that LLM agents can be a double-edged sword -- (1) they can work well when a high-quality plan and necessary user involvement in execution are available, and (2) users can easily mistrust the LLM agents with plans that seem plausible. We synthesized key insights for using LLM agents as daily assistants to calibrate user trust and achieve better overall task outcomes. Our work has important implications for the future design of daily assistants and human-AI collaboration with LLM agents.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Most AI Agents require explicit user instruction to act and lack the capacity to formulate or re-prioritize goals dynamically based on changing context/environment or evolving objectives [197].”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [197]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Plausible plans can induce inappropriate reliance [61].”
From Caring Without Feeling: Affective Dynamics as the Control Layer of Human-AI Agent Collaboration (2026) · cites this paper as [61]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Proactivity is similarly underdeveloped.”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [197]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical power analyses using G∗Power 3.1: tests for correlation and regression analyses
Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009) · Behavior Research Methods · doi:10.3758/brm.41.4.1149 · Cited by 37,615
The magical number seven, plus or minus two: Some limits on our capacity for processing information.
Miller, G. A. (1956) · Psychological Review · doi:10.1037/h0043158 · Cited by 17,812
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605
Loss aversion in riskless choice: A reference-dependent model.
Tversky, A., & Kahneman, D. (1991) · The Quarterly Journal of Economics · doi:10.2307/2937956 · Cited by 6,432 · cited 1×
“However, on tasks entailing potential risks ( e.g., monetary payments or hiring an employee), humans hesitate to trust such AI systems due to loss aversion [96] and algorithmic aversion [13, 19, 40, 71].”
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
Algorithm aversion: People erroneously avoid algorithms after seeing them err.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000033 · Cited by 3,232 · cited 2×
“However, on tasks entailing potential risks ( e.g., monetary payments or hiring an employee), humans hesitate to trust such AI systems due to loss aversion [96] and algorithmic aversion [13, 19, 40, 71].”
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Proceedings of the 13th International Conference on Neural Information Processing Systems
Leen, T. K., Dietterich, T. G., & Tresp, V. (2000) · Cited by 2,190
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187 · cited 1×
“With this in the backdrop, existing work on automated task completion has revealed that LLM agents can exhibit promising performance in handling complex tasks like playing games [105], answering complex questions [116], and in simulating social behavior [78].”
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · arXiv · doi:10.48550/arxiv.2304.03442 · Cited by 2,173
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090
Algorithm appreciation: People prefer algorithmic to human judgment
Logg, J. M., Minson, J. A., & Moore, D. A. (2019) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2018.12.005 · Cited by 2,061
Probabilistic machine learning and artificial intelligence
Ghahramani, Z. (2015) · Nature · doi:10.1038/nature14541 · Cited by 1,995 · cited 1×
“However, such AI systems are not good at dealing with out-of-distribution data [44, 72], and their intrinsic probabilistic nature brings much uncertainty in practice [28].”
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941
A Survey on Large Language Model based Autonomous Agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.11432 · Cited by 1,925
A survey on deep learning and its applications
Dong, S., Wang, P., & Abbas, K. (2021) · Computer Science Review · doi:10.1016/j.cosrev.2021.100379 · Cited by 1,800
Trust, control strategies and allocation of function in human-machine systems
LEE, J., & MORAY, N. (1992) · Ergonomics · doi:10.1080/00140139208967392 · Cited by 1,459
Adversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410 · cited 1×
“However, such AI systems are not good at dealing with out-of-distribution data [44, 72], and their intrinsic probabilistic nature brings much uncertainty in practice [28].”
A Survey on Deep Learning: Algorithms, Techniques, and Applications
Pouyanfar, S., Sadiq, S., Yan, Y., Tian, H., Tao, Y., Reyes, M. P., Shyu, M.-L., Chen, S.-C., & Iyengar, S. S. (2019) · ACM Computing Surveys · doi:10.1145/3234150 · Cited by 1,403
Task complexity: Definition of the construct
Wood, R. E. (1986) · Organizational Behavior and Human Decision Processes · doi:10.1016/0749-5978(86)90044-0 · Cited by 1,393 · cited 1×
“Based on existing literature on task complexity [85, 107], we considered component complexity to inform our selection.”
The role of trust in automation reliance
Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., Pierce, L. G., & Beck, H. P. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00038-7 · Cited by 1,276 · cited 1×
“Trust and reliance have been important research topics since human adoption of automation systems [17, 56].”
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2018) · Management Science · doi:10.1287/mnsc.2016.2643 · Cited by 1,230 · cited 1×
“Only when users can obtain a sense of con-trol by being able to modify the outcomes of imperfect AI can they overcome such algorithm aversion and be willing to collaborate with imperfect AI systems [14].”
Developing Theory Through Simulation Methods
Davis, J. P., Eisenhardt, K. M., & Bingham, C. B. (2007) · Academy of Management Review · doi:10.5465/amr.2007.24351453 · Cited by 1,157
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972 · cited 4×
“The term ‘ LLM agent ’ refers to an artificial entity based on LLMs that perceives its context, makes decisions, and then takes actions in response [109].”
InfluentialThe rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
McCoy, R. T., Pavlick, E., & Linzen, T. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1334 · Cited by 944
Zhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895
Trustworthy Artificial Intelligence: A Review
Kaur, D., Uslu, S., Rittichier, K. J., & Durresi, A. (2023) · ACM Computing Surveys · doi:10.1145/3491209 · Cited by 709
What influences algorithmic decision-making? A systematic literature review on algorithm aversion
Mahmud, H., Islam, A. K. M. N., Ahmed, S. I., & Smolander, K. (2022) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2021.121390 · Cited by 601
Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI
Jacovi, A., Marasović, A., Miller, T., & Goldberg, Y. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445923 · Cited by 545 · cited 1×
“Users with an uncalibrated trust in the AI system can be easily misled to disuse or misuse AI systems [42].”
Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems
(2025) · doi:10.1145/3706598 · Cited by 534
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527 · cited 3×
“Empirical studies have shown that most users tend to trust AI systems that are perceived to be highly accurate [111].”
Sharma, A., Lin, I. W., Miner, A. S., Atkins, D. C., & Althoff, T. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-022-00593-2 · Cited by 500
Wu, T., Terry, M., & Cai, C. J. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517582 · Cited by 483
Fügener, A., Grahl, J., Gupta, A., & Ketter, W. (2022) · Information Systems Research · doi:10.1287/isre.2021.1079 · Cited by 478
Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance
Bansal, G., Nushi, B., Kamar, E., Lasecki, W. S., Weld, D. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5285 · Cited by 466 · cited 1×
“Similarly, task characteristics like task complexity and uncertainty [85, 86] and factors of the AI system ( e.g., performance feedback [4, 65], AI transparency [100], stated accuracy [33], and confidence of AI advice [95, 112]) also affect user trust and reliance on the AI system.”
Kepuska, V., & Bohouta, G. (2018) · 2018 IEEE 8th Annual Computing and Communication Workshop and Conference (CCWC) · doi:10.1109/ccwc.2018.8301638 · Cited by 418
Autonomous agents modelling other agents: A comprehensive survey and open problems
Albrecht, S. V., & Stone, P. (2018) · Artificial Intelligence · doi:10.1016/j.artint.2018.01.002 · Cited by 417 · cited 1×
“Autonomous agents have been regarded as a research focus for artificial intelligence (AI) over the last century [1].”
Explanations Can Reduce Overreliance on AI Systems During Decision-Making
Vasconcelos, H., Jörke, M., Grunde-McLaughlin, M., Gerstenberg, T., Bernstein, M. S., & Krishna, R. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579605 · Cited by 402
Deep Learning for Medical Anomaly Detection – A Survey
Fernando, T., Gammulle, H., Denman, S., Sridharan, S., & Fookes, C. (2022) · ACM Computing Surveys · doi:10.1145/3464423 · Cited by 399
Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models
Wang, L., Xu, W., Lan, Y., Hu, Z., Lan, Y., Lee, R. K.-W., & Lim, E.-P. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.147 · Cited by 344
Wang, X., & Yin, M. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450650 · Cited by 320 · cited 2×
“For example, increasing the transparency of AI systems by providing confidence scores [112], explanations [104], trustworthiness cues [58], and uncertainty communication [50].”
Theoretical Considerations and Development of a Questionnaire to Measure Trust in Automation
Körber, M. (2019) · Advances in Intelligent Systems and Computing · doi:10.1007/978-3-319-96074-6_2 · Cited by 317 · cited 3×
“…we measured their perceived cognitive load using the NASA-TLX questionnaire [11], their overall trust in the daily assistant using the trust in automation questionnaire [51], and we gathered their feedback on our system (related to planning, execution, and other aspects) using open-ended text.”
InfluentialOlson, J. S., & Kellogg, W. A. (2014) · Springer eBooks · doi:10.1007/978-1-4939-0378-8 · Cited by 273
LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks
Kambhampati, S., Valmeekam, K., Guan, L., Verma, M., Stechly, K., Bhambri, S., Saldyt, L., & Murthy, A. (2024) · arXiv · doi:10.48550/arxiv.2402.01817 · Cited by 237
How to Evaluate Trust in AI-Assisted Decision Making? A Survey of Empirical Methodologies
Vereschak, O., Bailly, G., & Caramiaux, B. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476068 · Cited by 229
A survey on deep learning for software engineering
Yang, Y., Xia, X., Lo, D., & Grundy, J. (2022) · ACM Computing Surveys · doi:10.1145/3505243 · Cited by 223 · cited 1×
“In recent decades, deep learning-based AI systems have shown promising performance across various domains [23, 110] and applications [15, 81].”
Designing transparency for effective Human-AI collaboration
Vössing, M., Kühl, N., Lind, M., & Satzger, G. (2022) · Information Systems Frontiers · doi:10.1007/s10796-022-10284-3 · Cited by 223
Deciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-making
Rastogi, C., Zhang, Y., Wei, D., Varshney, K. R., Dhurandhar, A., & Tomsett, R. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512930 · Cited by 216 · cited 1×
“User factors like AI literacy [8], cognitive bias [82], peer input [84], and risk perception [24, 29] have been observed to substantially impact user trust and reliance on the AI system.”
Colligan, L., Potts, H. W. W., Finn, C. T., & Sinkin, R. A. (2015) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2015.03.003 · Cited by 214
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap
Liao, Q. V., & Wortman Vaughan, J. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.8036d03b · Cited by 210
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap
Liao, Q. V., & Vaughan, J. W. (2023) · arXiv · doi:10.48550/arxiv.2306.01941 · Cited by 209 · cited 1×
“Due to the capability of generating coherent, knowledgeable, and high-quality responses to diverse human input [106], a wide community of human-computer interaction researchers has paid attention to large language models [59].”
Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations
Schemmer, M., Kuehl, N., Benz, C., Bartos, A., & Satzger, G. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584066 · Cited by 208 · cited 1×
“Such user reliance patterns are denoted as appropriate reliance [87, 88], which is the key to achieving complementary team performance.”
Lai, V., Chen, C., Smith-Renner, A., Liao, Q. V., & Tan, C. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594087 · Cited by 204 · cited 6×
“The variables and measures used in our study refer to existing empirical studies of human-AI collaboration [53].”
InfluentialRapid Trust Calibration through Interpretable and Uncertainty-Aware AI
Tomsett, R., Preece, A., Braines, D., Cerutti, F., Chakraborty, S., Srivastava, M., Pearson, G., & Kaplan, L. (2020) · Patterns · doi:10.1016/j.patter.2020.100049 · Cited by 186 · cited 1×
“Similarly, task characteristics like task complexity and uncertainty [85, 86] and factors of the AI system ( e.g., performance feedback [4, 65], AI transparency [100], stated accuracy [33], and confidence of AI advice [95, 112]) also affect user trust and reliance on the AI system.”
Designing for Responsible Trust in AI Systems: A Communication Perspective
Liao, Q. V., & Sundar, S. S. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533182 · Cited by 165
Kim, S. S. Y., Liao, Q. V., Vorvoreanu, M., Ballard, S., & Vaughan, J. W. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658941 · Cited by 147
Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation
Lai, V., Carton, S., Bhatnagar, R., Liao, Q. V., Zhang, Y., & Tan, C. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501999 · Cited by 145
I Think I Get Your Point, AI! The Illusion of Explanatory Depth in Explainable AI
Chromik, M., Eiband, M., Buchner, F., Krüger, A., & Butz, A. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450644 · Cited by 144 · cited 2×
“Another example is that users can develop an illusion of explanatory depth brought by explainable AI techniques [10], which leads to uncalibrated trust in the AI system.”
Mehrotra, S., Degachi, C., Vereschak, O., Jonker, C. M., & Tielman, M. L. (2024) · ACM Journal on Responsible Computing · doi:10.1145/3696449 · Cited by 144
Ma, S., Lei, Y., Wang, X., Zheng, C., Shi, C., Yin, M., & Ma, X. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581058 · Cited by 143 · cited 1×
“Different techniques to help users realize the trustworthiness of the AI system have been proposed [48, 68, 83].”
Human Reliance on Machine Learning Models When Performance Feedback is Limited: Heuristics and Risks
Lu, Z., & Yin, M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445562 · Cited by 131
Human-LLM Collaborative Annotation Through Effective Verification of LLM Labels
Wang, X., Kim, H., Rahman, S., Mitra, K., & Miao, Z. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641960 · Cited by 128
Explaining machine learning models with interactive natural language conversations using TalkToModel
Slack, D., Krishna, S., Lakkaraju, H., & Singh, S. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00692-8 · Cited by 125
"Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humans
Lai, V., Liu, H., & Tan, C. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376873 · Cited by 123
Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistant
He, G., Aishwarya, N., & Gadiraju, U. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712133 · Cited by 115
Self-citationLiu, H., Lai, V., & Tan, C. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479552 · Cited by 111
Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems
He, G., Kuiper, L., & Gadiraju, U. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581025 · Cited by 107 · cited 4×
“Another example is tutorial interventions, which have shown effectiveness in user onboarding [54], mitigating cognitive biases [34] and developing AI literacy [8].”
InfluentialSelf-citationHou, Y. T.-Y., & Jung, M. F. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479864 · Cited by 105
Huang, S., Zhong, W., Lu, J., et al. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2401.17167 · Cited by 105
Rechkemmer, A., & Yin, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501967 · Cited by 97
Hemmer, P., Westphal, M., Schemmer, M., Vetter, S., Vössing, M., & Satzger, G. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584052 · Cited by 93
If in a Crowdsourced Data Annotation Pipeline, a GPT-4
He, Z., Huang, C.-Y., Ding, C. C., Rohatgi, S., & Huang, T.-H. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642834 · Cited by 93
Clarity is a Worthwhile Quality: On the Role of Task Clarity in Microtask Crowdsourcing
Gadiraju, U., Yang, J., & Bozzon, A. (2017) · Proceedings of the 28th ACM Conference on Hypertext and Social Media · doi:10.1145/3078714.3078715 · Cited by 87
Self-citationComplementarity in Human-AI Collaboration: Concept, Sources, and EvidencePreprint
Hemmer, P., Schemmer, M., Kühl, N., Vössing, M., & Satzger, G. (2024) · Repository KITopen (Karlsruhe Institute of Technology) · doi:10.48550/arxiv.2404.00029 · Cited by 82
The role of decision confidence in advice-taking and trust formation.
Pescetelli, N., & Yeung, N. (2021) · Journal of Experimental Psychology: General · doi:10.1037/xge0000960 · Cited by 73
Human Perceptions on Moral Responsibility of AI: A Case Study in AI-Assisted Bail Decision-Making
Lima, G., Grgić-Hlača, N., & Cha, M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445260 · Cited by 70
Capable but Amoral? Comparing AI and Human Expert Collaboration in Ethical Decision Making
Tolmeijer, S., Christen, M., Kandul, S., Kneer, M., & Bernstein, A. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517732 · Cited by 70
Green, B., & Chen, Y. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479562 · Cited by 69 · cited 2×
“User factors like AI literacy [8], cognitive bias [82], peer input [84], and risk perception [24, 29] have been observed to substantially impact user trust and reliance on the AI system.”
Second Chance for a First Impression? Trust Development in Intelligent System Interaction
Tolmeijer, S., Gadiraju, U., Ghantasala, R., Gupta, A., & Bernstein, A. (2021) · Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization · doi:10.1145/3450613.3456817 · Cited by 67 · cited 2×
“Combined with existing work on algorithm aversion [13] and the impact of negative first impressions on user trust [94], we can infer that such convincingly wrong content [91] can bias user trust and reliance towards the extremes.”
Self-citationAI Knowledge: Improving AI Delegation through Human Enablement
Pinski, M., Adam, M., & Benlian, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580794 · Cited by 62 · cited 2×
“Many user factors like trust [66], human expertise domain [20], and AI knowledge [80]) have a substantial impact on human delegation behaviors.”
Ma, S., Wang, X., Lei, Y., Shi, C., Yin, M., & Ma, X. (2024) · arXiv · doi:10.48550/arxiv.2403.09552 · Cited by 56
Banovic, N., Yang, Z., Ramesh, A., & Liu, A. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579460 · Cited by 55 · cited 1×
“As found by Banovic et al. [3], untrustworthy AI systems can deceive end users to gain their trust.”
Wang, Z., Cai, S., Chen, G., Liu, A., Ma, X., & Liang, Y. (2023) · arXiv · doi:10.48550/arxiv.2302.01560 · Cited by 55
You’d Better Stop! Understanding Human Reliance on Machine Learning Models under Covariate Shift
Chiang, C.-W., & Yin, M. (2021) · 13th ACM Web Science Conference 2021 · doi:10.1145/3447535.3462487 · Cited by 53 · cited 2×
“Researchers have proposed various interventions to promote appropriate reliance [8, 9, 31, 34, 64, 65] and calibrate user trust in AI systems [7, 112].”
Balayn, A., Yurrita, M., Rancourt, F., Casati, F., & Gadiraju, U. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713787 · Cited by 50
Self-citationA Missing Piece in the Puzzle: Considering the Role of Task Complexity in Human-AI Decision Making
Salimzadeh, S., He, G., & Gadiraju, U. (2023) · Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization · doi:10.1145/3565472.3592959 · Cited by 42 · cited 3×
“Based on existing literature on task complexity [85, 107], we considered component complexity to inform our selection.”
Self-citationShen, H., Huang, C.-Y., Wu, T., & Huang, T.-H. K. (2023) · Computer Supported Cooperative Work and Social Computing · doi:10.1145/3584931.3607492 · Cited by 42 · cited 2×
“Researchers have actively explored how LLMs can assist users in various tasks like data annotation [35, 103], programming [77], writing [5, 90], and fact verification [91].”
Evaluating Human-AI Partnership for LLM-based Code Migration
Omidvar-Tehrani, B., Ishaani, M., & Anubhai, A. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650896 · Cited by 42
Salimzadeh, S., He, G., & Gadiraju, U. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641905 · Cited by 41 · cited 3×
“In our study, all tasks are executed in a simulation setup, which has been a popular method for orchestrating meaningful human-centered AI studies [16, 86].”
Self-citationHe, G., Buijsman, S., & Gadiraju, U. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610067 · Cited by 39
Self-citationDoes More Advice Help? The Effects of Second Opinions in AI-Assisted Decision Making
Lu, Z., Wang, D., & Yin, M. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3653708 · Cited by 39
Understanding Choice Independence and Error Types in Human-AI Collaboration
Erlei, A., Sharma, A., & Gadiraju, U. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641946 · Cited by 38
Self-citationCrowdCO-OP: Sharing Risks and Rewards in Crowdsourcing
Fan, S., Gadiraju, U., Checco, A., & Demartini, G. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415203 · Cited by 34
Self-citationDecision-Oriented Dialogue for Human-AI Collaboration
Lin, J., Tomlin, N., Andreas, J., & Eisner, J. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00679 · Cited by 34
Erlei, A., Das, R., Meub, L., Anand, A., & Gadiraju, U. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517734 · Cited by 33
Self-citationHemmer, P., Schellhammer, S., Vössing, M., Jakubik, J., & Satzger, G. (2022) · Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/344 · Cited by 29
Large Language Models Help Humans Verify Truthfulness – Except When They Are Convincingly Wrong
Si, C., Goyal, N., Wu, T., Zhao, C., Feng, S., Daumé Iii, H., & Boyd-Graber, J. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.81 · Cited by 26
To Err Is AI! Debugging as an Intervention to Facilitate Appropriate Reliance on AI Systems
He, G., Bharos, A., & Gadiraju, U. (2024) · ACM Conference on Hypertext and Social Media (HT) · doi:10.1145/3648188.3675130 · Cited by 24
Self-citationFogliato, R., Chouldechova, A., & Lipton, Z. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479572 · Cited by 23
Synergizing Human-AI Agency: A Guide of 23 Heuristics for Service Co-Creation with LLM-Based AgentsPreprint
Zheng, Q., Xu, Z., Choudhry, A., Chen, Y.-T., Li, Y., & Huang, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.15065 · Cited by 21
Concept-Guided LLM Agents for Human-AI Safety Codesign
Geißler, F., Roscher, K., & Trapp, M. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v3i1.31188 · Cited by 20
Biswas, S., Erlei, A., & Gadiraju, U. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713201 · Cited by 20
Self-citationSalimzadeh, S., & Gadiraju, U. (2024) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3627043.3659567 · Cited by 19
Self-citationHuang, S., Zhong, W., Lu, J., Zhu, Q., Gao, J., Liu, W., Hou, Y., Zeng, X., Wang, Y., Shang, L., et al. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.259 · Cited by 15
He, G., Balayn, A., Buijsman, S., Yang, J., & Gadiraju, U. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15118 · Cited by 10
Self-citationMa, S., Chen, Q., Wang, X., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.16812 · Cited by 9
A Survey of AI ReliancePreprint
Eckhardt, S., Kühl, N., Dolata, M., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.03948 · Cited by 2
Jiang, J. A., Wade, K., Fiesler, C., & Brubaker, J. R. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449168 · Cited by 1
Research works printed but not linked: 9
- Chun-Wei Chiang and Ming Yin. 2022. Exploring the Effects of Machine Learning Literacy Interventions on Laypeople’s Reliance on Machine Learning Models. In IUI 2022: 27th International Conference on Intelligent User Interfaces, Helsinki, Finland, March 22 - 25, 2022, Giulio Jacucci, Samuel Kaski, Cristina Conati, Simone Stumpf, Tuukka Ruotsalo, and Krzysztof Gajos (Eds.). ACM, 148–161.
- Patrick Hemmer Max Schemmer Michael Vössing and Niklas Kühl. 2021. Human-AI Complementarity in Hybrid Intelligence Systems: A Structured Literature Review. PACIS (2021) 78.
- Patricia K Kahr Gerrit Rooks Martijn C Willemsen and Chris CP Snijders. 2024. Understanding Trust and Reliance Development in AI Advice: Assessing Model Accuracy Model Explanations and Experiences from Previous Interactions. ACM Transactions on Interactive Intelligent Systems (2024).
- Zhuoyan Li and Ming Yin. [n. d.]. Utilizing Human Behavior Modeling to Manipulate Explanations in AI-Assisted Decision Making: The Good, the Bad, and the Scary. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
- Brian Lubars and Chenhao Tan. 2019. Ask not what AI can do but what AI should do: Towards a framework of task delegability. Advances in neural information processing systems 32 (2019).
- David Madras Toni Pitassi and Richard Zemel. 2018. Predict responsibly: improving fairness and accuracy by learning to defer. Advances in neural information processing systems 31 (2018).
- Harikrishna Narasimhan Wittawat Jitkrittum Aditya K Menon Ankit Rawat and Sanjiv Kumar. 2022. Post-hoc estimators for learning to defer to an expert. Advances in Neural Information Processing Systems 35 (2022) 29292–29304.
- Max Schemmer, Patrick Hemmer, Niklas Kühl, Carina Benz, and Gerhard Satzger. 2022. Should I Follow AI-based Advice? Measuring Appropriate Reliance in Human-AI Decision-Making. In ACM Conference on Human Factors in Computing Systems (CHI’22), Workshop on Trust and Reliance in AI-Human Teams (trAIt).
- Yuchen Zhuang Yue Yu Kuan Wang Haotian Sun and Chao Zhang. 2024. Toolqa: A dataset for llm question answering with external tools. Advances in Neural Information Processing Systems 36 (2024).
Cited by
AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2025) · arXiv · doi:10.48550/arxiv.2505.10468 · Cited by 295
“Most AI Agents require explicit user instruction to act and lack the capacity to formulate or re-prioritize goals dynamically based on changing context/environment or evolving objectives [197].”
AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103599 · Cited by 671
“Most AI Agents require explicit user instruction to act and lack the capacity to formulate or re-prioritize goals dynamically based on changing context/environment or evolving objectives [197].”
AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i3.15161 · Cited by 186
Zheng, W., & Arakawa, Y. (2025) · IEEE Access · doi:10.1109/access.2025.3598088 · Cited by 13
“This study finds broad applications in more complicated and multiple tasks, such as agents [171], [172], [173], [174], assistive robots [175], [176], [177], [178], etc.”
Dong, A., Li, P., Chen, Y., Gibson, S., Zhao, L., & He, M. (2026) · Sustainability · doi:10.3390/su18115313 · Cited by 9
Metadata
- Authors
- He, G., Demartini, G., & Gadiraju, U.
- Affiliations
- Delft University of Technology (NL)
- The University of Queensland (AU)
- Venue
- Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2025-04-25
- Pages
- 1-22
- OpenAlex
- W4407170313
- Cited by
- 23
- References
- 96
- Citations by year
- 2026: 15 · 2025: 8
- FWCI
- 43.92
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Australian Research Council (FT240100022, IC200100022)
- Topics
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Keywords
- LLM agents, user trust, team performance, sequential decision making, human-in-the-loop planning, human-AI collaboration, user participation, airline reservations, credit card payments, simulation environment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.