Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts
Open accessThe DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index; 18 of the 33 sources its claims cite were checked for support.
Abstract
Pre-trained large language models (“LLMs”) like GPT-3 can engage in fluent, multi-turn instruction-taking out-of-the-box, making them attractive materials for designing natural language interactions. Using natural language to steer LLM outputs (“prompting”) has emerged as an important design technique potentially accessible to non-AI-experts. Crafting effective prompts can be challenging, however, and prompt-based interactions are brittle. Here, we explore whether non-AI-experts can successfully engage in “end-user prompt engineering” using a design probe—a prototype LLM-based chatbot design tool supporting development and systematic evaluation of prompting strategies. Ultimately, our probe participants explored prompt designs opportunistically, not systematically, and struggled in ways echoing end-user programming systems and interactive machine learning systems. Expectations stemming from human-to-human instructional experiences, and a tendency to overgeneralize, were barriers to effective prompt design. These findings have implications for non-AI-expert-facing LLM-based tool design and for improving LLM-and-prompt literacy among programmers and the public, and present opportunities for further research.
Overview
CoChat Summary
TL;DR
This design-probe study found that 10 non-AI experts could iterate on LLM prompts opportunistically but not systematically, and their struggles echoed end-user programming and interactive machine learning barriers. Barriers included overgeneralizing from single successes/failures and applying human-to-human social expectations to prompt design.
Key findings
- All 10 participants engaged in ad hoc, opportunistic prompt iteration with local testing; two out of ten needed assistance to begin, but then could iterate on their own.
- Zero participants used BotDesigner’s systematic prompt-testing interface for the prompt design task, and none elicited more than one or two conversations before jumping in to fix errors.
- Participants overgeneralized from single observations, often declaring success after one correct response or assuming incapability after one failed prompt; all held a strong bias for direct instruction over in-line examples, and only P5 independently added example dialog to the preamble.
- Human-human interaction expectations shaped behavior: some mixed instructions to the bot with instructions to the bot’s user, expected memory across conversations, used polite phrasing such as “please,” and avoided repetition even when encouraged.
Methods
- The authors built BotDesigner, a no-code React/node.js prompt-design probe using OpenAI’s GPT-3 `text-davinci-002` with temperature set to 0, supporting prompt-template authoring, conversation testing, error labeling, and systematic replay.
- They conducted a think-aloud user study with 10 participants with little to no prompt-design experience, asking them to improve a baseline instructional chatbot (“CarlaBot”) toward behaviors from a chef transcript, with up to 1 hour for the task.
- Analysis involved transcribing videos, exploratory data analysis, affinity diagrams, and a service blueprint to compare prompt-design approaches and struggles across participants.
Limitations
- Stated: the participant pool was small and skewed toward professionals and graduate students in STEM-related fields, likely early adopters, so findings are not claimed to be universal or complete; the time given for prompt design was relatively limited.
- Stated: interviewer advice was not intentionally varied across participants, and because participants focused on different errors, not all received the same advice or advice in the same order.
- Evident from the text: the study used one tool, one underlying model, and a chatbot domain, so the authors note open questions about generalization to other domains and broader populations.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
12 Valid · 6 Not valid · 8 Not checked · 83 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“Prompt design tools to date have focused on supporting professional programmers [45] and NLP practitioners [42], rather than non-AI experts, non-programmers, and other potential end-users of these systems.”
§ Why Johnny Can't Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts > 1 INTRODUCTION
What the sources say
- Disputedcites [45] 45. Hendrik Strobelt, Albert Webson, Victor Sanh, Benjamin Hoover, Johanna Beyer, Hanspeter Pfster, and Alexander M Rush. 2022. Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models. arXiv preprint arXiv:2208.07852 (2022). sourceExperimental — accuracy not yet measured
The citation is attached to “NLP practitioners,” but the paper describes PromptIDE as a tool for domain experts, not one focused on NLP practitioners instead of end-users. The Introduction (p. 2) says it supports prompt construction for domain experts, and the Conclusions (p. 9) say those users need no training expertise. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
- Partly supportedcites [42] 42. Victor Sanh, Albert Webson, Colin Rafel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chafn, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, sourceExperimental — accuracy not yet measured
Appendix C describes PromptSource as a tool for writing prompts for NLP datasets, using Jinja templates and Python-dictionary examples. That supports citing this paper as an example of an NLP-oriented prompt-design tool, but it does not explicitly identify NLP practitioners as its intended users or establish that such tools generally exclude non-experts and other end-users. (LLM, conf=0.77)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Recent pre-trained LLMs (e.g., GPT-3 [8], ChatGPT [1]) can engage in fuent, multi-turn conversations outof-the-box”
§ Why Johnny Can't Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts > 1 INTRODUCTION
What the sources say
- Partly supportedcites [8] 8. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877-1901. sourceExperimental — accuracy not yet measured
The citation to this paper is attached to GPT-3, not ChatGPT. Section 3.6 (p. 18) reports strong GPT-3 performance on a conversational question-answering dataset, and the Introduction (p. 5) says it can perform tasks without fine-tuning. Those findings do not establish that GPT-3 can engage in fluent, multi-turn conversations out of the box; the reported conversational result is in a few-shot setting. (LLM, conf=0.83)
- Not matchedcites [1] 1. 2022. CHATGPT: Optimizing language models for dialogue. https://openai.com/ blog/chatgpt/
We searched by its title and found no matching paper.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Today's chatbot design practice-i.e., designing multi-turn conversational interactions-follows a well-established workfow [12,13,24,25,39,44].”
§ Why Johnny Can't Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts > 2 RELATED WORK 2.1 The Promises of Non-Expert Prompt Design
What the sources say
- Partly supportedcites [12] 12. Zhifa Chen, Yichen Lu, Mika P. Nieminen, and Andrés Lucero. 2020. Creating a Chatbot for and with Migrants: Chatbot Personality Drives Co-Design Activities. Association for Computing Machinery, New York, NY, USA, 219-230. https: //doi.org/10.1145/3357236.3395495 sourceExperimental — accuracy not yet measured
The Participants and Procedure section describes designing a chatbot persona and sketching conversation flows, so the paper supports those elements of the claimed process. It does not establish that this is a well-established workflow across today's chatbot practice or describe filling dialogue templates with supervised NLP models. The Context section describes its implemented prototype as a one-turn question-answering system, rather than a multi-turn implementation. (LLM, conf=0.85)
- Partly supportedcites [13] 13. Justin Cranshaw, Emad Elwany, Todd Newman, Rafal Kocielnik, Bowen Yu, Sandeep Soni, Jaime Teevan, and Andrés Monroy-Hernández. 2017. Calendar. help: Designing a workfow-based scheduling agent with humans in the loop. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 2382-2393. sourceExperimental — accuracy not yet measured
The Discussion and Tier 1: Automatic Microtask Execution sections describe Calendar.help’s iterative development of workflows and use of ML and NLP. They do not establish a well-established chatbot-design workflow involving persona selection, dialogue-flow templates, and supervised models. Page numbers are unavailable in our copy. (LLM, conf=0.82)
- Not checked — no full text of the sourcecites [24] 24. Bogyeong Kim, Jaehoon Pyun, and Woohun Lee. 2018. Enhancing Storytelling Experience with Story-Aware Interactive Puppet. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI EA '18). ACM, New York, NY, USA, Article LBW076, 6 pages. https://doi.org/10.1145/3170427.3188515 source
- Not checked — no full text of the sourcecites [25] 25. Scott R. Klemmer, Anoop K. Sinha, Jack Chen, James A. Landay, Nadeem Aboobaker, and Annie Wang. 2000. Suede: A Wizard of Oz Prototyping Tool for Speech User Interfaces. In Proceedings of the 13th Annual ACM Symposium on User Interface Software and Technology (San Diego, California, USA) (UIST '00). ACM, New York, NY, USA, 1-10. https://doi.org/10.1145/354401.354406 source
- Not checked — no full text of the sourcecites [39] 39. Catherine Pricilla, Dessi Puji Lestari, and Dody Dharma. 2018. Designing inter- action for chatbot-based conversational commerce with user-centered design. In 2018 5th International Conference on Advanced Informatics: Concept Theory and Applications (ICAICTA). IEEE, 244-249. source
- Not checked — no full text of the sourcecites [44] 44. Lisa Stifelman, Adam Elman, and Anne Sullivan. 2013. Designing Natural Speech Interactions for the Living Room. In CHI '13 Extended Abstracts on Human Factors in Computing Systems (Paris, France) (CHI EA '13). ACM, New York, NY, USA, 1215-1220. https://doi.org/10.1145/2468356.2468574 source
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“substantially lowering the data and programming-skill barriers to creating passable conversational user experiences [7].”
§ Why Johnny Can't Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts > 1 INTRODUCTION
What the sources say
- Partly supportedcites [7] 7. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Dem- szky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle sourceExperimental — accuracy not yet measured
Section 2.5, Interaction (p. 44), says language-based foundation models could significantly lower the application-development threshold by reducing the need to collect data or train models from scratch, allowing even non-ML experts to prototype applications. It does not establish that these applications are specifically conversational or that they produce passable conversational user experiences; it also frames the reduction as a possibility rather than an established result. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Expectations stemming from human-to-human instructional experiences, and a tendency to overgeneralize, were barriers to efective prompt design.”
- Also stated in: § 1 INTRODUCTION · plain
“Additional barriers to efective prompt design stem from limited conceptions of LLMs' prompt understanding and execution abilities and their understandable inclinations to design prompts that resemble human-to-human instructions.”
- Also stated in: § 4 USER STUDY FINDINGS · plain
“models of system behavior rooted in human-human interactions.”
§ Why Johnny Can't Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts > Abstract
plainstates a causen = 10- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 INTRODUCTION · plain
45 more claims
Create a free account to see all 109 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 38
- It may state two separate claims as one. 28
- The quote only partly states this claim. 2
- Borderline: one of the tests for a substantial claim is unclear. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Ubiquitous performance degradation over multi-turn interactions is likely a reason for low uptake of AI systems [73, 4, 28], particularly with novice users who are less skilled at providing complete, detailed instructions from the onset of conversation [87, 35].”
From LLMs Get Lost In Multi-Turn Conversation (2025) · cites this paper as [87]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Zamfirescu et al. [57] found that users unfamiliar with PE tend to overgeneralize from single failures (causing them to throw out potentially good prompts), rather than having a holistic view of the overall performance of a prompt or chain.”
From Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences (2024) · cites this paper as [57]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Additionally, LLM prompt design is difficult for novice users [90]”
From Homogenization Effects of Large Language Models on Human Creative Ideation (2024) · cites this paper as [90]
The cited source says what the claim attributes to it.
In the citing paper
“Novice users also face significant hurdles in prompt engineering [24, 26].”
From Scaffolding Creativity: Integrating Generative AI Tools and Real-World Experiences in Business Education (2025) · cites this paper as [24]
The cited source says what the claim attributes to it.
In the citing paper
“Although prompting has been widely studied in computer science (Debnath et al. 2025), research in non-expert settings remains limited (Dierickx et al. 2026;Lee and Palmer 2025;Zamfirescu-Pereira et al. 2023).”
From A Taxonomy of LLM-Supported Tasks for Critical AI Literacy in Journalism (2026) · cites this paper as Zamfirescu-Pereira et al. 2023
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 3×
“The emergent “pre-train, prompt, predict 1 ” paradigm in NLP promises to lower the entry barrier for non-experts innovating on conversational interactions [30].”
BackgroundOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Zero-shot text-to-image generation
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021) · arXiv · doi:10.48550/arxiv.2102.12092 · Cited by 2,417
Nass, C., Steuer, J., & Tauber, E. R. (1994) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/191666.191703 · Cited by 2,148 · cited 2×
“” (P6)”
The computer for the 21 st century
Weiser, M. (1999) · ACM SIGMOBILE Mobile Computing and Communications Review · doi:10.1145/329124.329126 · Cited by 1,234
Service Blueprinting: A Practical Technique for Service Innovation
Bitner, M. J., Ostrom, A. L., & Morgan, F. N. (2008) · California Management Review · doi:10.2307/41166446 · Cited by 1,194
Beyond the five-user assumption: benefits of increased sample sizes in usability testing
Faulkner, L. (2003) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03195514 · Cited by 1,053 · cited 1×
“Our sample size was chosen in line with prior work around formative testing for usability [19, 43]: our goal is to explore the frst, “easy-to-fnd” problems users encounter when engaged in our task, and our experience with a pilot user suggested that problem discoverability was likely to be high.”
MethodQuestioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925 · cited 1×
“[29] and iterative dialog approaches by Lakkaraju et al.”
MethodReexamining the cluster hypothesis: scatter/gather on retrieval results
Hearst, M. A., & Pedersen, J. O. (1996) · Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR '96 · doi:10.1145/243199.243216 · Cited by 822
Moggridge, B. (2006) · doi:10.1075/idj.17.1.16ehr · Cited by 795
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.301 · Cited by 693
Design Guidelines for Prompt Engineering Text-to-Image Generative Models
Liu, V., & Chilton, L. B. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501825 · Cited by 639
Design Guidelines for Prompt Engineering Text-to-Image Generative Models
Liu, V., & Chilton, L. B. (2021) · arXiv · doi:10.48550/arxiv.2109.06977 · Cited by 635 · cited 1×
“describe a study with artists, identifying emergent strategies for prototyping prompts, such as “try multiple generations to get a representative idea of what prompts return” and “focus on keywords rather than the phrasings of the prompt” [31].”
BackgroundThe state of the art in end-user software engineering
Ko, A. J., Abraham, R., Beckwith, L., Blackwell, A., Burnett, M., Erwig, M., Scaffidi, C., Lawrance, J., Lieberman, H., Myers, B., et al. (2011) · ACM Computing Surveys · doi:10.1145/1922649.1922658 · Cited by 633
Multitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612 · cited 13×
“Designers frst (i) identify the chatbot’s functionality or persona and draft ideal user-bot conversations, for example, through Wizard-of-Oz or having experts drafting scripts; (ii) create a dialogue fow template (e.g., “(1) greeting message; (2) questions to collect user intention; (3) ...”); (iii) fll the template w…”
BackgroundInfluentialNass, C., Steuer, J., & Tauber, E. R. (1994) · Conference companion on Human factors in computing systems - CHI '94 · doi:10.1145/259963.260288 · Cited by 579
Design galleries: a general approach to setting parameters for computer graphics and animation
Marks, J., Ruml, W., Ryall, K., Seims, J., Shieber, S., Andalman, B., Beardsley, P. A., Freeman, W., Gibson, S., Hodgins, J., et al. (1997) · Proceedings of the 24th annual conference on Computer graphics and interactive techniques - SIGGRAPH '97 · doi:10.1145/258734.258887 · Cited by 551
Six Learning Barriers in End-User Programming Systems
Ko, A. J., Myers, B. A., & Aung, H. H. (2004) · 2004 IEEE Symposium on Visual Languages - Human Centric Computing · doi:10.1109/vlhcc.2004.47 · Cited by 549 · cited 5×
“…then assumed that no instructions of that type would succeed , that the bot was incapable of following the requested instruction, regardless of phrasing , leading to an early exit from a successfully selected prompt type that was merely being misused: “ok, I guess it doesn’t like to tell jokes.””
BackgroundInfluentialBoehner, K., Vertesi, J., Sengers, P., & Dourish, P. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240789 · Cited by 525 · cited 1×
“• Conversation authoring & interaction : Users instruct chatbot behavior with prompts alone (without code) and can observe the efects of their prompts by engaging in conversation with the chatbot.”
BackgroundWu, T., Terry, M., & Cai, C. J. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517582 · Cited by 483 · cited 1×
“In this work, we use these previously-identifed challenges to better understand why we observe some of the struggles with end-user prompt engineering that we do.”
MethodA Review of User Interface Design for Interactive Machine Learning
Dudley, J. J., & Kristensson, P. O. (2018) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3185517 · Cited by 343 · cited 1×
“How can BotDesigner enable users to inspect a prompt’s impact on a full set of conversations, while each of these conversations unfolds diferently depending on the user’s utterances and the LLM’s probabilistic outputs? To address the challenge in tracking prompt’s efects, BotDesigner builds on tool design in prior int…”
MethodSauro, J., & Lewis, J. R. (2016) · Quantifying the User Experience · doi:10.1016/b978-0-12-802308-2.00002-3 · Cited by 312
Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language ModelsPreprint
Logan, R. L., IV, Balavzevi'c, I., Wallace, E., Petroni, F., Singh, S., & Riedel, S. (2021) · Findings · doi:10.48550/arxiv.2106.13353 · Cited by 253
Suede: a Wizard of Oz prototyping tool for speech user interfaces
Klemmer, S. R., Sinha, A. K., Chen, J., Landay, J. A., Aboobaker, N., & Wang, A. (2000) · Proceedings of the 13th annual ACM symposium on User interface software and technology - UIST '00 · doi:10.1145/354401.354406 · Cited by 230
Seamful design: showing the seams in wearable computing
Chalmers, M. (2003) · IEE Eurowearable '03 · doi:10.1049/ic:20030140 · Cited by 206 · cited 1×
“””
Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation With Large Language Models
Strobelt, H., Webson, A., Sanh, V., Hoover, B., Beyer, J., Pfister, H., & Rush, A. M. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209479 · Cited by 204 · cited 2×
“Prompt design tools to date have focused on supporting professional programmers [45] and NLP practitioners [42], rather than non-AI experts, non-programmers, and other potential end-users of these systems.”
BackgroundInteractive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models
Strobelt, H., Webson, A., Sanh, V., Hoover, B., Beyer, J., Pfister, H., & Rush, A. M. (2022) · arXiv · doi:10.48550/arxiv.2208.07852 · Cited by 204 · cited 2×
“Prompt design tools to date have focused on supporting professional programmers [45] and NLP practitioners [42], rather than non-AI experts, non-programmers, and other potential end-users of these systems.”
BackgroundPromptChainer: Chaining Large Language Model Prompts through Visual Programming
Wu, T., Jiang, E., Donsbach, A., Gray, J., Molina, A., Terry, M., & Cai, C. J. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519729 · Cited by 198 · cited 2×
“In this work, we use these previously-identifed challenges to better understand why we observe some of the struggles with end-user prompt engineering that we do.”
BackgroundMethodGrounding Interactive Machine Learning Tool Design in How Non-Experts Actually Build Models
Yang, Q., Suh, J., Chen, N.-C., & Ramos, G. (2018) · Proceedings of the 2018 Designing Interactive Systems Conference · doi:10.1145/3196709.3196729 · Cited by 156 · cited 4×
“We want to dive deep into what those struggles reveal about people’s assumptions and understanding of prompt-based systems in order to inform how end-user-facing prompt design tools might best support their users.”
BackgroundInfluentialSelf-citationCarter, S., Armstrong, Z., Schubert, L., Johnson, I., & Olah, C. (2019) · Distill · doi:10.23915/distill.00015 · Cited by 147 · cited 1×
“However, we did validate BotDesigner’s functionality and usability throughout this workfow via a set of pilot user studies (see Appendix A.3 for details) to ensure that these pieces are available should users choose to systematically assess their prompt designs.”
MethodCutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models
Logan IV, R., Balazevic, I., Wallace, E., Petroni, F., Singh, S., & Riedel, S. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.222 · Cited by 133
Prevent-Teach-Reinforce: A Standardized Model of School-Based Behavioral Intervention
Dunlap, G., Iovannone, R., Wilson, K. J., Kincaid, D. K., & Strain, P. (2010) · Journal of Positive Behavior Interventions · doi:10.1177/1098300708330880 · Cited by 124
Calendar.help: Designing a Workflow-Based Scheduling Agent with Humans in the Loop
Cranshaw, J., Elwany, E., Newman, T., Kocielnik, R., Yu, B., Soni, S., Teevan, J., & Monroy-Hernández, A. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025780 · Cited by 106
Dang, H., Mecke, L., Lehmann, F., Goller, S., & Buschek, D. (2022) · arXiv · doi:10.48550/arxiv.2209.01390 · Cited by 92 · cited 1×
“However, it is unclear whether the resulting strategies are efective or robust beyond the few interaction contexts that the designers experimented with.”
Background"Beautiful Seams": Strategic Revelations and Concealments
Inman, S., & Ribes, D. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300508 · Cited by 88
Calendar.Help: Designing a Workflow-Based Scheduling Agent with Humans in the LoopPreprint
Cranshaw, J., Kocielnik, R., Yu, B., & Teevan, J. B. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2940295 · Cited by 67
Rethinking Explainability as a Dialogue: A Practitioner's Perspective
Lakkaraju, H., Slack, D., Chen, Y., Tan, C., & Singh, S. (2022) · arXiv · doi:10.48550/arxiv.2202.01875 · Cited by 62
Creating a Chatbot for and with Migrants: Chatbot Personality Drives Co-Design Activities
Chen, Z., Lu, Y., Nieminen, M. P., & Lucero, A. (2020) · Proceedings of the 2020 ACM Designing Interactive Systems Conference · doi:10.1145/3357236.3395495 · Cited by 61 · cited 1×
“Today’s chatbot design practice—i.e., designing multi-turn conversational interactions—follows a well-established workfow [12, 13, 24, 25, 39, 44].”
BackgroundOpenPrompt: An Open-source Framework for Prompt-learning
Ding, N., Hu, S., Zhao, W., Chen, Y., Liu, Z., Zheng, H.-T., & Sun, M. (2021) · arXiv · doi:10.48550/arxiv.2111.01998 · Cited by 60 · cited 2×
“In this paradigm, people without programming skills or NLP knowledge can nonetheless make NLP models generate desired interactions by crafting efective prompt strategies (e.g., natural language instructions, examples, and templates) [3, 15].”
BackgroundStory Centaur: Large Language Model Few Shot Learning as a Creative Writing Tool
Swanson, B., Mathewson, K., Pietrzak, B., Chen, S., & Dinalescu, M. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2021.eacl-demos.29 · Cited by 53 · cited 1×
“Here, we briefy overview known barriers and challenges in end-user programming and iML (list below [27]) as well as how experts and non-experts approach these tasks diferently (Table 2), to motivate and contextualize our investigation into end-users’ intuitive approaches to prompt design.”
BackgroundDesigning Interaction for Chatbot-Based Conversational Commerce with User-Centered Design
Pricilla, C., Lestari, D. P., & Dharma, D. (2018) · 2018 5th International Conference on Advanced Informatics: Concept Theory and Applications (ICAICTA) · doi:10.1109/icaicta.2018.8541320 · Cited by 48 · cited 1×
“, designing multi-turn conversational interactions—follows a well-established workfow [12, 13, 24, 25, 39, 44].”
Background(2017) · Digital Design in Action · doi:10.1201/9781315734224-4 · Cited by 46
GenLine and GenForm: Two Tools for Interacting with Generative Language Models in a Code Editor
Jiang, E., Toh, E., Molina, A., Donsbach, A., Cai, C. J., & Terry, M. (2021) · Adjunct Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3474349.3480209 · Cited by 27 · cited 2×
“This excitement around LLMs and prompting is propelling a rapidly growing set of LLM-powered applications [23] and prompt design tools [3, 20, 32].”
BackgroundGANzilla: User-Driven Direction Discovery in Generative Adversarial Networks
Evirgen, N., & Chen, X. '. (2022) · Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3526113.3545638 · Cited by 22 · cited 2×
“Modern takes on this work have traded CGI parameters for deep model hyperpa-rameters, such as Carter et al. ’s Activation Atlas [10] for inspecting image classifers, Evirgen and Chen’s GANzilla [18] for searching a GAN-based image generator’s latent space using scatter/gather interactions [21], and Zhang and Banovic’s…”
MethodPromptSource: An Integrated Development Environment and Repository for Natural Language Prompts
Bach, S. H., Sanh, V., Yong, Z.-X., Webson, A., Raffel, C., Nayak, N. V., Sharma, A., Kim, T., Bari, M. S., Fevry, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2202.01279 · Cited by 21
Designing natural speech interactions for the living room
Stifelman, L., Elman, A., & Sullivan, A. (2013) · CHI '13 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/2468356.2468574 · Cited by 13 · cited 1×
“, designing multi-turn conversational interactions—follows a well-established workfow [12, 13, 24, 25, 39, 44].”
BackgroundA Natural Language Programming System for Text Processing
Barnett, M. P., & Ruhsam, W. M. (1968) · IEEE Transactions on Engineering Writing and Speech · doi:10.1109/tews.1968.4322334 · Cited by 12 · cited 1×
“The idea of instructing computers in natural language has fascinated researchers for decades, as it promises to make the power of computing more customizable and accessible to people without programming training [4].”
BackgroundEnhancing Storytelling Experience with Story-Aware Interactive Puppet
Kim, B., Pyun, J., & Lee, W. (2018) · Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3170427.3188515 · Cited by 7 · cited 1×
“Today’s chatbot design practice—i.e., designing multi-turn conversational interactions—follows a well-established workfow [12, 13, 24, 25, 39, 44].”
BackgroundRealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · arXiv · doi:10.48550/arxiv.2009.11462
Research works printed but not linked: 1
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.). Vol. 33. Curran Associates, Inc., 1877–1901.
Cited by
MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots
Deng, G., Liu, Y., Li, Y., Wang, K., Zhang, Y., Li, Z., Wang, H., Zhang, T., & Liu, Y. (2023) · arXiv · doi:10.48550/arxiv.2307.08715 · Cited by 418
“Prompt engineering [51], [53], [34] plays an instrumental role in the development of language models, providing a means to significantly augment a model’s ability to undertake tasks it has not been directly trained for.”
BackgroundChatGPT improves creative problem-solving performance in university students: An experimental study
Urban, M., Děchtěrenko, F., Lukavský, J., Hrabalová, V., Svacha, F., Brom, C., & Urban, K. (2024) · Computers & Education · doi:10.1016/j.compedu.2024.105031 · Cited by 396
AI literacy and its implications for prompt engineering strategies
Knoth, N., Tolzin, A., Janson, A., & Leimeister, J. M. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100225 · Cited by 368
“Zamfirescu-Pereira et al. (2023) examined the potential of prompt engineering by non-experts, using a prototype LLM-based chatbot design tool.”
BackgroundMethodInfluentialUsing an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 487
Using large language models in psychology
Demszky, D., Yang, D., Yeager, D. S., Bryan, C. J., Clapper, M., Chandhok, S., Eichstaedt, J. C., Hecht, C., Jamieson, J., Johnson, M., et al. (2023) · Nature Reviews Psychology · doi:10.1038/s44159-023-00241-5 · Cited by 369
“Thus, prompt engineering (in which users vary the prompts given to the LLM to achieve a better output for a given purpose) is a key aspect of prompt-tuning that requires subtle expertise 30,31 .”
Background
Metadata
- Authors
- Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q.
- Affiliations
- University of California, Berkeley (US)
- Georgia Institute of Technology (US)
- EECS, UC Berkeley, United States
- Cornell University (US)
- Venue
- Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2023-04-19
- Pages
- 1-21
- OpenAlex
- W4366548330
- Cited by
- 1,274
- References
- 37
- Citations by year
- 2026: 484 · 2025: 407 · 2024: 319 · 2023: 63
- FWCI
- 144.17
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by-nc)
- Funders
- USAF/DARPA (FA8750-20-C-0156, FA8750-20-C-0074, and FA8750-20-C0155 (SDCPS Program))
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- prompt engineering, large language models, non-expert users, end-user programming, chatbot design, design probes, overgeneralization, natural language interfaces, human-computer interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-06, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-06 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.