Is It AI or Is It Me? Understanding Users’ Prompt Journey with Text-to-Image Generative AI Tools
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
Generative Artificial Intelligence (AI) has witnessed unprecedented growth in text-to-image AI tools. Yet, much remains unknown about users’ prompt journey with such tools in the wild. In this paper, we posit that designing human-centered text-to-image AI tools requires a clear understanding of how individuals intuitively approach crafting prompts, and what challenges they may encounter. To address this, we conducted semi-structured interviews with 19 existing users of a text-to-image AI tool. Our findings (1) offer insights into users’ prompt journey including structures and processes for writing, evaluating, and refining prompts in text-to-image AI tools and (2) indicate that users must overcome barriers to aligning AI to their intents, and mastering prompt crafting knowledge. From the findings, we discuss the prompt journey as an individual yet a social experience and highlight opportunities for aligning text-to-image AI tools and users’ intents.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Resnick, M., Maloney, J., Monroy-Hernández, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J., Silverman, B., et al. (2009) · Communications of the ACM · doi:10.1145/1592761.1592779 · Cited by 3,333
What is AI Literacy? Competencies and Design Considerations
Long, D., & Magerko, B. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376727 · Cited by 3,241 · cited 1×
“Following established human-AI interaction guidelines [5, 34, 45], text-to-image tools should explicitly communicate both strengths and limitations, emphasizing the inherent imperfection and unpredictability of results [56, 59, 63].”
Self-citationGuidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 2×
“Following established human-AI interaction guidelines [5, 34, 45], text-to-image tools should explicitly communicate both strengths and limitations, emphasizing the inherent imperfection and unpredictability of results [56, 59, 63].”
Newborn infants imitate adult facial gestures
Meltzoff, A. N., & Moore, M. K. (1983) · Child Development · doi:10.2307/1130058 · Cited by 1,307
Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts
Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581388 · Cited by 1,274 · cited 2×
“A noteworthy example is the work by Zamfirescu-Pereira et al. [68], which explores intuitive prompting behaviors with large language models with a conversation agent.”
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
Reynolds, L., & McDonell, K. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3451760 · Cited by 952 · cited 3×
“As a result, users of text-to-image AI tools and researchers have developed guidelines and documentation in the form of handbooks [1, 3, 19, 53] and tutorials [38, 65] to address the ambiguity of crafting prompts within these tools, a process commonly referred to as ‘prompt engineering’ [47].”
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Lu, Y., Bartolo, M., Moore, A., Riedel, S., & Stenetorp, P. (2021) · arXiv · doi:10.48550/arxiv.2104.08786 · Cited by 615 · cited 1×
“Prior research on prompt-based interactions has examined factors influencing the effectiveness of prompts in large language models, such as the negative impact of lengthy instructions [42] or the role of word order in achieving accurate model responses [36].”
Multitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612 · cited 1×
“It has since evolved to denote the meticulous process of selecting the right prompt to produce intended outputs with large language models [51].”
Denny, P., Kumar, V., & Giacaman, N. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569823 · Cited by 287 · cited 2×
“Prior work has systematically evaluated the effectiveness of different approaches to prompting in diffusion models and proposed various prompt engineering guidelines and taxonomies [16, 32, 64].”
Novice-AI Music Co-Creation via AI-Steering Tools for Deep Generative Models
Louie, R., Coenen, A., Huang, C. Z., Terry, M., & Cai, C. J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376739 · Cited by 271 · cited 1×
“Challenges of content generation with AI include the open-ended nature of generative outputs, lack of user agency, misalignment with users’ creative goals, and information overload [10, 35, 48].”
Investigating Explainability of Generative AI for Code through Scenario-based Design
Sun, J., Liao, Q. V., Muller, M., Agarwal, M., Houde, S., Talamadupula, K., & Weisz, J. D. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511119 · Cited by 219 · cited 1×
“Following established human-AI interaction guidelines [5, 34, 45], text-to-image tools should explicitly communicate both strengths and limitations, emphasizing the inherent imperfection and unpredictability of results [56, 59, 63].”
Large-scale Text-to-Image Generation Models for Visual Artists’ Creative Works
Ko, H.-K., Park, G., Jeon, H., Jo, J., Kim, J., & Seo, J. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584078 · Cited by 189
Hughes, R. T., Zhu, L., & Bednarz, T. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.604234 · Cited by 178
Design Ideation with AI - Sketching, Thinking and Talking with Generative Machine Learning Models
Tholander, J., & Jonsson, M. (2023) · Proceedings of the 2023 ACM Designing Interactive Systems Conference · doi:10.1145/3563657.3596014 · Cited by 148
Perfection Not Required? Human-AI Partnerships in Code Translation
Weisz, J. D., Muller, M., Houde, S., Richards, J., Ross, S. I., Martinez, F., Agarwal, M., & Talamadupula, K. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450656 · Cited by 129
Design implications of generative AI systems for visual storytelling for young learners
Han, A., & Cai, Z. (2023) · Proceedings of the 22nd Annual ACM Interaction Design and Children Conference · doi:10.1145/3585088.3593867 · Cited by 127 · cited 1×
“A similar emphasis on stakeholder collaboration is noted by Han and Cai [22].”
Liu, M. X., Sarkar, A., Negreanu, C., Zorn, B., Williams, J., Toronto, N., & Gordon, A. D. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580817 · Cited by 119
Designing with AI: An Exploration of Co-Ideation with Image Generators
Chiou, L.-Y., Hung, P.-K., Liang, R.-H., & Wang, C.-T. (2023) · Proceedings of the 2023 ACM Designing Interactive Systems Conference · doi:10.1145/3563657.3596001 · Cited by 113
Designing Co-Creative AI for Public Spaces
Long, D., Jacob, M., & Magerko, B. (2019) · Proceedings of the 2019 on Creativity and Cognition · doi:10.1145/3325480.3325504 · Cited by 70 · cited 1×
“These challenges are further compounded in specialized settings like public spaces [33], virtual environments [59], and game design [45], each bringing their specific challenges and requirements.”
Self-citationToward General Design Principles for Generative AI Applications
Weisz, J. D., Muller, M., He, J., & Houde, S. (2023) · arXiv · doi:10.48550/arxiv.2301.05578 · Cited by 54
Research works printed but not linked: 28
- Ash Amin and Joanne Roberts. 2008. Knowing in action: Beyond communities of practice. Research policy 37, 2 (2008), 353–369.
- Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77–101.
- John Seely Brown and Paul Duguid. 1991. Organizational learning and communities-of-practice: Toward a unified view of working, learning, and innovation. Organization science 2, 1 (1991), 40–57.
- 10.1177/1478077120934126 · DOI
- 10.1145/3591196.3593515 · DOI · Paper page
- 10.1145/3586183.3606777 · DOI · Paper page
- 10.1016/j.ijhcs.2023.103029 · DOI
- Nassim Dehouche and Kullathida Dehouche. 2023. What’s in a text-to-image prompt? The potential of stable diffusion in visual arts education. Heliyon (2023).
- Daniel Dewey. 2014. Reinforcement learning and the reward engineering principle. In 2014 AAAI Spring Symposium Series.
- 10.1145/3544549.3585737 · DOI · Paper page
- Matthew Guzdial and Mark Riedl. 2019. An interaction framework for studying co-creative ai. arXiv preprint arXiv:1903.09709 (2019). · arXiv
- Ellen Jiang, Kristen Olson, Edwin Toh, Alejandra Molina, Aaron Donsbach, Michael Terry, and Carrie J Cai. 2022. PromptMaker: Prompt-based Prototyping with Large Language Models. In CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1–8.
- Ellen Jiang, Edwin Toh, Alejandra Molina, Kristen Olson, Claire Kayacik, Aaron Donsbach, Carrie J Cai, and Michael Terry. 2022. Discovering the Syntax and Strategies of Natural Language Programming with Generative Language Models. In CHI Conference on Human Factors in Computing Systems. 1–19.
- Vivian Liu and Lydia B Chilton. 2022. Design Guidelines for Prompt Engineering Text-to-Image Generative Models. In CHI Conference on Human Factors in Computing Systems. 1–23.
- 10.18653/v1 · DOI
- 10.19173/irrodl.v14i5.1579 · DOI
- Joe O’Connor and Jacob Andreas. 2021. What Context Features Can Transformer Language Models Use?arXiv preprint arXiv:2106.08367 (2021). · arXiv
- 10.1080/0144929X.2023.2286532 · DOI · Paper page
- 10.1145/3472538.3472573 · DOI
- Jeba Rezwana and Mary Lou Maher. 2022. Designing Creative AI Partners with COFI: A Framework for Modeling Interaction in Human-AI Co-Creative Systems. ACM Transactions on Computer-Human Interaction (2022).
- 10.1145/3591196.3593051 · DOI · Paper page
- 10.1145/3531146.3534639 · DOI · Paper page
- Josh Urban Davis Fraser Anderson Merten Stroetzel Tovi Grossman and George Fitzmaurice. 2021. Designing co-creative ai for virtual environments. In Creativity and Cognition. 1–11. · DOI · Paper page
- 10.1007/s10676-017-9440-6 · DOI
- Veera Vimpari, Annakaisa Kultima, Perttu Hämäläinen, and Christian Guckelsberger. 2023. " An Adapt-or-Die Type of Situation": Perception, Adoption, and Use of Text-To-Image-Generation AI by Game Industry Professionals. arXiv preprint arXiv:2302.12601 (2023). · arXiv
- Sam Witteveen and Martin Andrews. 2022. Investigating Prompt Engineering in Diffusion Models. arXiv preprint arXiv:2211.15462 (2022). · arXiv
- Xiao Xie, Xiwen Cai, Junpei Zhou, Nan Cao, and Yingcai Wu. 2018. A semantic-based method for visualizing large image collections. IEEE transactions on visualization and computer graphics 25, 7 (2018), 2362–2377.
- Daijin Yang, Yanpeng Zhou, Zhiyuan Zhang, Toby Jia-Jun Li, and Ray LC. 2022. AI as an Active Writer: Interaction strategies with generated text in human-AI collaborative fiction writing. In Joint Proceedings of the ACM IUI Workshops 2022, Vol. 10.
Cited by
AIdeation: Designing a Human-AI Collaborative Ideation System for Concept Designers
Wang, W.-F., Lu, C.-T., Ponsa i Campanyà, N., Chen, B.-Y., & Chen, M. Y. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714148 · Cited by 50
Zhang, S., Wang, H., & Yi, X. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757594 · Cited by 36
Bennett, C. L., Kane, S. K., & Harrington, C. N. (2025) · Proceedings of the 5th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3757887.3763012 · Cited by 3
Alharbi, R., Cheong, A. D., Herskovitz, J., Brewer, R. N., & Schoenebeck, S. (2025) · Proceedings of the 27th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3663547.3746376 · Cited by 3
“Generally, there is a wealth of HCI and AI scholarship dedicated to understanding how sighted users recognize and resolve errors (e.g., [32, 44, 61, 67, 75]).”
Raees, M., Iwamoto, Y., Papangelis, K., Heard, J., & Tigwell, G. W. (2026) · arXiv · doi:10.48550/arxiv.2602.08937 · Cited by 2
“For instance, research examines accessibility guidelines [35, 81], the difficulty in writing alt text [45, 79, 73], and the emerging role of AI in supporting authors [46, 66].”
Metadata
- Authors
- Mahdavi Goloujeh, A., Sullivan, A., & Magerko, B.
- Venue
- Proceedings of the CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2024-05-11
- Pages
- 1-13
- OpenAlex
- W4396833147
- Cited by
- 89
- References
- 20
- FWCI
- 29.45
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.