CoChat
Sign Up
PreprintCited by 8Speech and dialogue systemsSocial Robot Interaction and HRIUsability and User Interface Design

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

Challenges in Human-Agent Communication

Open access
Bansal, Gagan, Vaughan, Jennifer Wortman, Amershi, Saleema, Horvitz, Eric, Fourney, Adam, Mozannar, Hussein, Dibia, Victor, Weld, Daniel S.·2024·arXiv
DOI 10.48550/arxiv.2412.10380arXiv 2412.10380
Open this paper in CoChat — free accountOpen PDF

Abstract

Remarkable advancements in modern generative foundation models have enabled the development of sophisticated and highly capable autonomous agents that can observe their environment, invoke tools, and communicate with other agents to solve problems. Although such agents can communicate with users through natural language, their complexity and wide-ranging failure modes present novel challenges for human-AI interaction. Building on prior research and informed by a communication grounding perspective, we contribute to the study of \emph{human-agent communication} by identifying and analyzing twelve key communication challenges that these systems pose. These include challenges in conveying information from the agent to the user, challenges in enabling the user to convey information to the agent, and overarching challenges that need to be considered across all human-agent communication. We illustrate each challenge through concrete examples and identify open directions of research. Our findings provide insights into critical gaps in human-agent communication research and serve as an urgent call for new design patterns, principles, and guidelines to support transparency and control in these systems.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2412.10380

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 64

  1. Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, et al. Guidelines for human-ai interaction. In CHI, 2019.
  2. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016. · arXiv
  3. Dario Amodei, Christopher Olah, Jacob Steinhardt, Paul Francis Christiano, John Schul- man, and Dandelion Mané. Concrete problems in ai safety. ArXiv, abs/1606.06565, 2016. · arXiv
  4. Gagan Bansal, Besmira Nushi, Ece Kamar, Walter S. Lasecki, Daniel S. Weld, and Eric Horvitz. Beyond accuracy: The role of mental models in human-ai team performance. In AAAI Conference on Human Computation & Crowdsourcing, 2019.
  5. Carrie J. Cai, Samantha Winter, David Steiner, Lauren Wilcox, and Michael Terry. Hello ai: Uncovering the onboarding needs of medical practitioners for human-ai collaborative decision-making. In CSCW, 2019.
  6. Harold R. Carey and Patrick R. Laughlin. Groups perform better than the best indi- viduals on letters-to-numbers problems: Effects of induced strategies. Group Processes & Intergroup Relations, 15(2):231-242, 2012.
  7. Tathagata Chakraborti, Sarath Sreedharan, and Subbarao Kambhampati. The emerging landscape of explainable automated planning & decision making. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, 2020.
  8. Alan Chan, Rebecca Salganik, Alva Markelius, Chris Pang, Nitarshan Rajkumar, Dmitrii Krasheninnikov, Lauro Langosco, Zhonghao He, Yawen Duan, Micah Carroll, Michelle Lin, Alex Mayhew, Katherine Collins, Maryam Molamohammadi, John Burden, Wanru Zhao, Shalaleh Rismani, Konstantinos Voudouris, Umang Bhatt, Adrian Weller, David Krueger, and Tegan Maharaj. Harms from increasingly agentic algorithmic systems. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, 2023.
  9. Anamaria Crisan, Margaret Drouhard, Jesse Vig, and Nazneen Rajani. Interactive model cards: A human-centered approach to model documentation. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, 2022.
  10. Anind K. Dey and Jennifer Mankoff. Designing mediation for context-aware applications. ACM Trans. Comput. Hum. Interact., 12:53-80, 2005.
  11. Victor Dibia, Jingya Chen, Gagan Bansal, Suff Syed, Adam Fourney, Erkang Zhu, Chi Wang, and Saleema Amershi. Autogen studio: A no-code developer tool for building and debugging multi-agent systems. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 72-79, 2024.
  12. Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608, 2017. · arXiv
  13. Motahhare Eslami, Karrie Karahalios, Christian Sandvig, Kristen Vaccaro, Aimee Rick- man, Kevin Hamilton, and Alex Kirlik. First i "like" it, then i hide it: Folk theories of social feeds. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI '16, page 2371-2382, New York, NY, USA, 2016. Association for Computing Machinery.
  14. Daniel Weld Oren Etzioni and Daniel Weld. The first law of robotics (a call to arms). In Proceedings of the 12th National Conference on AI, volume 17, page 23, 1994.
  15. Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, et al. Magentic- one: A generalist multi-agent system for solving complex tasks. arXiv preprint arXiv:2411.04468, 2024. · arXiv
  16. Maria Fox, Derek Long, and Daniele Magazzeni. Explainable planning. arXiv preprint arXiv:1709.10256, 2017. · arXiv
  17. Iason Gabriel, Arianna Manzini, Geoff Keeling, Lisa Anne Hendricks, Verena Rieser, Hasan Iqbal, Nenad Tomašev, Ira Ktena, Zachary Kenton, Mikel Rodriguez, et al. The ethics of advanced ai assistants. arXiv preprint arXiv:2404.16244, 2024. · arXiv
  18. Charles Goodwin. Conversational Organization: Interaction Between Speakers and Hear- ers. Academic Press, New York, 1981.
  19. M. A. K. Halliday and Ruqaiya Hasan. Language, Context, and Text: Aspects of Language in a Social-Semiotic Perspective. Oxford University Press, Oxford, 1989.
  20. Gayle W Hill. Group versus individual performance: Are n+ 1 heads better than one? Psychological bulletin, 91(3):517, 1982.
  21. Eric Horvitz. Principles of mixed-initiative interfaces. In CHI, 1999.
  22. Eric Horvitz, Susan Dumais, and Paul Koch. Learning predictive models of memory land- marks. In Proceedings of the annual meeting of the cognitive science society, volume 26, 2004.
  23. Eric Horvitz and Tim Paek. A computational architecture for conversation. In UM99 User Modeling: Proceedings of the Seventh International Conference, pages 201-210. Springer, 1999.
  24. Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, Fanzhi Zeng, Kwan Yee Ng, Juntao Dai, Xuehai Pan, Aidan O'Gara, Yingshan Lei, Hua Xu, Brian Tse, Jie Fu, Stephen Marcus McAleer, Yaodong Yang, Yizhou Wang, Song-Chun Zhu, Yike Guo, and Wen Gao. Ai alignment: A comprehensive survey. ArXiv, abs/2310.19852, 2023. · arXiv
  25. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Delong Chen, Wenliang Dai, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55:1 -38, 2022.
  26. S. Kambhampati. Challenges of human-aware ai systems. AI Magazine, 41(3):3-17, 2020.
  27. Subbarao Kambhampati. Can large language models reason and plan? Annals of the New York Academy of Sciences, 1534(1):15-18, 2024.
  28. Subbarao Kambhampati, Sarath Sreedharan, Mudit Verma, Yantian Zha, and L. Guan. Symbols as a lingua franca for bridging human-ai chasm for explainable and advisable ai systems. In AAAI Conference on Artificial Intelligence, 2021.
  29. Kimiya Keyvan and Jimmy Xiangji Huang. How to approach ambiguous queries in conver- sational search: A survey of techniques, approaches, tools, and challenges. ACM Comput. Surv., 55(6), December 2022.
  30. Patrick R. Laughlin and John Adamopoulos. Social combination processes and individual learning for six-person cooperative groups on an intellective task. Journal of Personality and Social Psychology, 38:941-947, 1980.
  31. Patrick R Laughlin and Alan L Ellis. Demonstrability and social combination processes on mathematical intellective tasks. Journal of Experimental Social Psychology, 22(3):177-189, 1986.
  32. Seth Lazar. LM Agents: Prospects and Impacts. Tutorial at the 2024 ACM Conference on Fairness, Accountability, and Transparency, 2024.
  33. Toby Jia-Jun Li, Jingya Chen, Haijun Xia, Tom Michael Mitchell, and Brad A. Myers. Multi-modal repairs of conversational breakdowns in task-oriented dialogs. Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology, 2020.
  34. Q. Vera Liao and Kush R. Varshney. Human-centered explainable AI (XAI): From algo- rithms to user experiences. arXiv preprint arXiv:2110.10790, 2021. · arXiv
  35. Zachary C. Lipton. The mythos of model interpretability. Communications of the ACM, 61(10):36-43, 2018.
  36. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2023.
  37. Ewa Luger and Abigail Sellen. "like having a really bad pa": The gulf between user expec- tation and experience of conversational agents. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI '16, page 5286-5297, New York, NY, USA, 2016. Association for Computing Machinery.
  38. Michael A. Madaio, Luke Stark, Jennifer Wortman Vaughan, and Hanna M. Wallach. Co-designing checklists to understand organizational challenges and opportunities around fairness in ai. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 2020.
  39. Jennifer Mankoff, Scott E. Hudson, and Gregory D. Abowd. Interaction techniques for ambiguity resolution in recognition-based interfaces. In ACM Symposium on User Interface Software and Technology, 2000.
  40. Hussein Mozannar, Jimin Lee, Dennis Wei, Prasanna Sattigeri, Subhro Das, and David Sontag. Effective human-ai teams via learned natural language rules and onboarding. Advances in Neural Information Processing Systems, 36, 2024.
  41. Tim Paek and Eric Horvitz. Conversation as action under uncertainty. In Proceedings of the Sixteenth Conference on Uncertainty in Artificial Intelligence (UAI2000), UAI 2000, 2000.
  42. Tomislav Pejsa, Dan Bohus, Michael F. Cohen, Chit W. Saw, James Mahoney, and Eric Horvitz. Natural communication about uncertainties in situated interaction. In Proceedings of the 16th International Conference on Multimodal Interaction, ICMI '14, page 283-290, New York, NY, USA, 2014. Association for Computing Machinery.
  43. Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake M. Hofman, Jennifer Wortman Vaughan, and Hanna Wallach. Manipulating and measuring model interpretability. In Proceedings of the 2021 ACM CHI Conference on Human Factors in Computing Systems, 2021.
  44. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. ArXiv, abs/2305.18290, 2023. · arXiv
  45. Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Sprin- genberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley D. Edwards, Nicolas Manfred Otto Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. ArXiv, abs/2205.06175, 2022. · arXiv
  46. Meredith Ringel, Ed Cutrell, Susan Dumais, and Eric Horvitz. Milestones in time: The value of landmarks in retrieving information from personal stores. In Proceedings of Interact 2003, 2003.
  47. Stephanie Rosenthal, Sarjoun Skaff, Manuela Veloso, Dan Bohus, and Eric Horvitz. Ex- ecution memory for grounding and coordination. In 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 213-214. IEEE, 2013.
  48. Stuart Russell and Peter Norvig. Artificial Intelligence: A Modern Approach. Pearson, 4th edition, 2020.
  49. Stuart J Russell and Benjamin N Grosof. A declarative approach to bias in concept learning. In Proceedings of the sixth National conference on Artificial intelligence-Volume 2, pages 505-510, 1987.
  50. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36, 2024.
  51. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. ArXiv, abs/2302.04761, 2023. · arXiv
  52. Ben Shneiderman. Human-centered artificial intelligence: Reliable, safe & trustworthy. International Journal of Human-Computer Interaction, 36:495 -504, 2020.
  53. Yoav Shoham and Kevin Leyton-Brown. Multiagent Systems: Algorithmic, Game- Theoretic, and Logical Foundations. Cambridge University Press, 2009.
  54. Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambham- pati. On the planning abilities of large language models: A critical investigation. ArXiv, abs/2305.15771, 2023. · arXiv
  55. Remo M. A. van der Heiden, Shamsi T. Iqbal, and Christian P. Janssen. Priming drivers before handover in semi-autonomous cars. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, 2017.
  56. Helena Vasconcelos, Matthew Jörke, Madeleine Grunde-McLaughlin, Tobias Gerstenberg, Michael Bernstein, and Ranjay Krishna. Explanations can reduce overreliance on ai systems during decision-making. Proceedings of the ACM on Human-Computer Interaction, 7:1 - 38, 2022.
  57. Jennifer Wortman Vaughan and Hanna Wallach. A human-centered agenda for intelligible machine learning. In Marcello Pelillo and Teresa Scantamburlo, editors, Machines We Trust: Perspectives on Dependable AI. MIT Press, 2021.
  58. Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. Openhands: An open platform for ai software developers as generalist agents. arXiv preprint, 2024.
  59. Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory. arXiv preprint arXiv:2409.07429, 2024. · arXiv
  60. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022.
  61. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. AutoGen: Enabling Next-Gen LLM Applications via Multi- Agent Conversation. In ICLR Workshop on Large Language Model (LLM) Agents, 2024.
  62. Wei Xu, Marvin J. Dainoff, Liezhong Ge, and Zaifeng Gao. Transitioning to human inter- action with ai systems: New challenges and opportunities for hci professionals to enable human-centered ai. International Journal of Human-Computer Interaction, 39:494 -518, 2021.
  63. Qian Yang, Aaron Steinfeld, Carolyn Penstein Rosé, and John Zimmerman. Re-examining whether, why, and how human-ai interaction is uniquely difficult to design. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 2020.
  64. Nur Yildirim, Mahima Pushkarna, Nitesh Goyal, Martin Wattenberg, and Fernanda Vi'egas. Investigating how practitioners use human-ai guidelines: A case study on the people + ai guidebook. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 2023.

Cited by

Sorted by citations

  • Why Do Multi-Agent LLM Systems Fail?

    Cemri, M., Pan, M. Z., Yang, S., Agrawal, L. A., Chopra, B., Tiwari, R., Keutzer, K., Parameswaran, A., Klein, D., Ramchandran, K., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.13657 · Cited by 630

    “While Bansal et al. [28] catalogs challenges in human-agent interaction, our contribution focuses specifically on failures within autonomous MAS execution.”

Metadata

Authors
Bansal, G., Vaughan, J. W., Amershi, S., Horvitz, E., Fourney, A., Mozannar, H., Dibia, V., & Weld, D. S.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2412.10380
arXiv
2412.10380
OpenAlex
W4405467589
Cited by
8
References
1
Citations by year
2026: 4 · 2025: 4
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
Microsoft Research
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
Keywords
human-agent communication, autonomous agents, generative foundation models, human-AI interaction, information transmission, transparency, user control, design guidelines, failure modes

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review