Herding AI Cats: Lessons from Designing a Chatbot by Prompting GPT-3
Open accessThe DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
Prompting Large Language Models (LLMs) is an exciting new approach to designing chatbots. But can it improve LLM’s user experience (UX) reliably enough to power chatbot products? Our attempt to design a robust chatbot by prompting GPT-3/4 alone suggests: not yet. Prompts made achieving “80%” UX goals easy, but not the remaining 20%. Fixing the few remaining interaction breakdowns resembled herding cats: We could not address one UX issue or test one design solution at a time; instead, we had to handle everything everywhere all at once. Moreover, because no prompt could make GPT reliably say “I don’t know” when it should, the user-GPT conversations had no guardrails after a breakdown occurred, often leading to UX downward spirals. These risks incentivized us to design highly prescriptive prompts and scripted bots, counter to the promises of LLM-powered chatbots. This paper describes this case study, unpacks prompting’s fickleness and its impact on UX design processes, and discusses implications for LLM-based design methods and tools.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738
Re-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design
Yang, Q., Steinfeld, A., Rosé, C., & Zimmerman, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376301 · Cited by 622
Self-citationMultitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612 · cited 7×
“This is because prior research has shown the promises of such design processes for prompt design [34], and that some NLP, HCI, and UX knowledge is necessary [33].”
BackgroundInfluentialChatGPT and the future of medical writing
Biswas, S. (2023) · Radiology · doi:10.1148/radiol.223312 · Cited by 571 · cited 1×
“Despite these limitations, a wide range of LLM-and-prompting-based chatbots are rapidly emerging in HCI research and elsewhere [27, 28], including in high-stake domains such as medicine [16, 32], law [11], and education [5].”
BackgroundLee, M., Liang, P., & Yang, Q. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502030 · Cited by 345 · cited 1×
“Our dialogue annotation and error analyses included: • Basic text-generation quality metrics, such as factualness [15]; • The extent to which the best-performing prompt can enable a GPT bot to exhibit the four interaction qualities listed in §3.1.2; • The severity of the interaction breakdowns,…”
MethodInfluentialSelf-citationPersuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good
Wang, X., Shi, W., Kim, R., Oh, Y., Yang, S., Zhang, J., & Yu, Z. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1566 · Cited by 222 · cited 2×
“Today, designers craft chatbot behaviors via both direct control (e.g., designing hand-drawn dialogue trees, canned bot responses) and indirect control (e.g., designing neural-network that powers end-to-end conversations by designing its neural architecture and dataset [26]).”
BackgroundPromptChainer: Chaining Large Language Model Prompts through Visual Programming
Wu, T., Jiang, E., Donsbach, A., Gray, J., Molina, A., Terry, M., & Cai, C. J. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519729 · Cited by 198 · cited 1×
“Despite these limitations, a wide range of LLM-and-prompting-based chatbots are rapidly emerging in HCI research and elsewhere [27, 28], including in high-stake domains such as medicine [16, 32], law [11], and education [5].”
BackgroundSketching NLP: A Case Study of Exploring the Right Things To Design with Language Intelligence
Yang, Q., Cranshaw, J., Amershi, S., Iqbal, S. T., & Teevan, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300415 · Cited by 90 · cited 3×
“These restrictions have long thwarted chatbot UX [29].”
BackgroundMethodSelf-citationPromptSource: An Integrated Development Environment and Repository for Natural Language Prompts
Bach, S. H., Sanh, V., Yong, Z.-X., Webson, A., Raffel, C., Nayak, N. V., Sharma, A., Kim, T., Bari, M. S., Fevry, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2202.01279 · Cited by 21 · cited 1×
“, designing prompts that look somewhat like code [4] and including instructions and examples of desired interactions in the prompt [7, 23]).”
Background
Cited by
Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences
Shankar, S., Zamfirescu-Pereira, J. D., Hartmann, B., Parameswaran, A., & Arawjo, I. (2024) · Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3654777.3676450 · Cited by 151
“Beyond classical summarization and NLP tasks, in response to the ad-hoc tedium of PE [59], academics and developers are building automated prompt optimization tools, maximizing some user-defned metric on a labeled set of examples.”
Self-citationEvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria
Kim, T. S., Lee, Y., Shin, J., Kim, Y.-H., & Kim, J. (2023) · arXiv · doi:10.48550/arxiv.2309.13633 · Cited by 82
Automating the Development of Task-oriented LLM-based Chatbots
Sánchez Cuadrado, J., Pérez-Soler, S., Guerra, E., & De Lara, J. (2024) · ACM Conversational User Interfaces 2024 · doi:10.1145/3640794.3665538 · Cited by 46
Rambler: Supporting Writing With Speech via LLM-Assisted Gist Manipulation
Lin, S., Warner, J., Zamfirescu-Pereira, J. D., Lee, M. G., Jain, S., Cai, S., Lertvittayakumjorn, P., Huang, M. X., Zhai, S., Hartmann, B., et al. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642217 · Cited by 39
Self-citationPetridis, S., Wedin, B. D., Wexler, J., Pushkarna, M., Donsbach, A., Goyal, N., Cai, C. J., & Terry, M. (2024) · Proceedings of the 29th International Conference on Intelligent User Interfaces · doi:10.1145/3640543.3645144 · Cited by 31
Metadata
- Authors
- Zamfirescu-Pereira, J. D., Wei, H., Xiao, A., Gu, K., Jung, G., Lee, M. G., Hartmann, B., & Yang, Q.
- Venue
- Proceedings of the 2023 ACM Designing Interactive Systems Conference
- Publisher
- ACM
- Published
- 2023-07-10
- Pages
- 2206-2220
- OpenAlex
- W4383679928
- Cited by
- 69
- References
- 10
- FWCI
- 13.04
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Herding AI Cats: Lessons from Designing a Chatbot by Prompting GPT-3. https://app.cochat.ai/papers/10.1145/3563657.3596138
- The paper
- Zamfirescu-Pereira, J. D., Wei, H., Xiao, A., Gu, K., Jung, G., Lee, M. G., Hartmann, B., & Yang, Q. (2023). Herding AI Cats: Lessons from Designing a Chatbot by Prompting GPT-3. Proceedings of the 2023 ACM Designing Interactive Systems Conference. https://doi.org/10.1145/3563657.3596138
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.