PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement
Open accessThe DOI matches its Crossref record; 74 of the 76 references it cites were found in a registry or an index.
Abstract
The recent advancements in Generative AI have significantly advanced the field of text-to-image generation. The state-of-the-art text-to-image model, Stable Diffusion, is now capable of synthesizing high-quality images with a strong sense of aesthetics. Crafting text prompts that align with the model’s interpretation and the user’s intent thus becomes crucial. However, prompting remains challenging for novice users due to the complexity of the stable diffusion model and the non-trivial efforts required for iteratively editing and refining the text prompts. To address these challenges, we propose PromptCharm, a mixed-initiative system that facilitates text-to-image creation through multi-modal prompt engineering and refinement. To assist novice users in prompting, PromptCharm first automatically refines and optimizes the user’s initial prompt. Furthermore, PromptCharm supports the user in exploring and selecting different image styles within a large database. To assist users in effectively refining their prompts and images, PromptCharm renders model explanations by visualizing the model’s attention values. If the user notices any unsatisfactory areas in the generated images, they can further refine the images through model attention adjustment or image inpainting within the rich feedback loop of PromptCharm. To evaluate the effectiveness and usability of PromptCharm, we conducted a controlled user study with 12 participants and an exploratory user study with another 12 participants. These two studies show that participants using PromptCharm were able to create images with higher quality and better aligned with the user’s expectations compared with using two variants of PromptCharm that lacked interaction or visualization support.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This differentiation is crucial when interacting with AI partners capable of autonomous action [3], where high-level user intent (agency) must interface with low-level system manipulation (control) [186].”
From Exploring Collaboration Patterns and Strategies in Human-AI Co-creation through the Lens of Agency: A Scoping Review of the Top-tier HCI Literature (2025) · cites this paper as [186]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892 · cited 1×
“The user interface of PromptCharm is implemented with Material UI 5 .”
MethodSelf-citationLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“For instance, few-shot prompting [6, 19, 79] and chain-of-thought prompting [68, 72, 74] are both representative prompting techniques for generative language models.”
BackgroundHigh-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302 · cited 5×
“As a result, SOTA text-to-image generation models (e.g., Stable Diffusion [53] and DALL-E2[49]) are mostly based on a pipeline of combining diffusion models with CLIP.”
BackgroundMethodInfluential"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 1×
“CLIP (contrast language-image pre-training) further enhances the diffusion model’s ability to understand both linguistic and visual concepts [48].”
BackgroundScikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539 · cited 1×
“(2”
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks
Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., & Metaxas, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.629 · Cited by 3,021 · cited 2×
“With the advancements in generative adversarial networks (GANs) [21], a large body of research on text-to-image generation has been focusing on GAN-based approaches [15, 33, 51, 78].”
BackgroundMethodTaming Transformers for High-Resolution Image Synthesis
Esser, P., Rombach, R., & Ommer, B. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01268 · Cited by 2,514 · cited 2×
“The emergence of the transformer [60] models further enhanced the GAN model’s capability of generating high-quality images [15].”
BackgroundTaming Transformers for High-Resolution Image Synthesis
Esser, P., Rombach, R., & Ommer, B. (2020) · arXiv · doi:10.48550/arxiv.2012.09841 · Cited by 2,514 · cited 2×
“The emergence of the transformer [60] models further enhanced the GAN model’s capability of generating high-quality images [15].”
BackgroundHierarchical Text-Conditional Image Generation with CLIP Latents
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2204.06125 · Cited by 2,511
Generative Adversarial Text to Image Synthesis
Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., & Lee, H. (2016) · arXiv · doi:10.48550/arxiv.1605.05396 · Cited by 2,494 · cited 2×
“With the advancements in generative adversarial networks (GANs) [21], a large body of research on text-to-image generation has been focusing on GAN-based approaches [15, 33, 51, 78].”
BackgroundMethodGuidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 1×
“As a result, the design of PromptCharm is highly motivated by the recent guidelines and design principles for human-AI collaboration [3, 7, 8, 14, 34, 61].”
Principles of mixed-initiative user interfaces
Horvitz, E. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303030 · Cited by 1,490 · cited 1×
“As a result, participants performed better when using PromptCharm to solve the close-ended tasks, which require them to replicate a particular image and its style.”
Making Pre-trained Language Models Better Few-shot Learners
Gao, T., Fisch, A., & Chen, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.295 · Cited by 1,380 · cited 1×
“For instance, few-shot prompting [6, 19, 79] and chain-of-thought prompting [68, 72, 74] are both representative prompting techniques for generative language models.”
BackgroundAutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., & Singh, S. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.346 · Cited by 1,279
Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts
Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581388 · Cited by 1,274 · cited 5×
“In addition to prompting techniques, there has also been a few studies about prompting guidelines [35, 36, 45, 77].”
BackgroundMethodInfluentialTree of Thoughts: Deliberate Problem Solving with Large Language Models
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2305.10601 · Cited by 1,158
LAION-5B: An open large-scale dataset for training next generation image-text models
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.08402 · Cited by 1,022 · cited 1×
“The Stable Diffusion model [53] was trained on the LAION dataset [54] with latent diffusion models and cross-attention layers.”
MethodQuestioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925 · cited 2×
“As a result, the design of PromptCharm is highly motivated by the recent guidelines and design principles for human-AI collaboration [3, 7, 8, 14, 34, 61].”
BackgroundMethodHigh-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737 · cited 5×
“As a result, SOTA text-to-image generation models (e.g., Stable Diffusion [53] and DALL-E2[49]) are mostly based on a pipeline of combining diffusion models with CLIP.”
BackgroundMethodInfluentialVector Quantized Diffusion Model for Text-to-Image Synthesis
Gu, S., Chen, D., Bao, J., Wen, F., Zhang, B., Chen, D., Yuan, L., & Guo, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01043 · Cited by 720
Design Guidelines for Prompt Engineering Text-to-Image Generative Models
Liu, V., & Chilton, L. B. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501825 · Cited by 639 · cited 7×
“(2”
BackgroundMethodInfluentialDesign Guidelines for Prompt Engineering Text-to-Image Generative Models
Liu, V., & Chilton, L. B. (2021) · arXiv · doi:10.48550/arxiv.2109.06977 · Cited by 635 · cited 7×
“(2”
BackgroundMethodInfluentialCai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359206 · Cited by 594 · cited 1×
“As a result, the design of PromptCharm is highly motivated by the recent guidelines and design principles for human-AI collaboration [3, 7, 8, 14, 34, 61].”
What Makes Good In-Context Examples for GPT-3?
Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., & Chen, W. (2022) · Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures · doi:10.18653/v1/2022.deelio-1.10 · Cited by 563
Design galleries: a general approach to setting parameters for computer graphics and animation
Marks, J., Ruml, W., Ryall, K., Seims, J., Shieber, S., Andalman, B., Beardsley, P. A., Freeman, W., Gibson, S., Hodgins, J., et al. (1997) · Proceedings of the 24th annual conference on Computer graphics and interactive techniques - SIGGRAPH '97 · doi:10.1145/258734.258887 · Cited by 551
The future of interactive systems and the emergence of direct manipulation†
SHNEIDERMAN, B. (1982) · Behaviour & Information Technology · doi:10.1080/01449298208914450 · Cited by 548 · cited 1×
“This is highly motivated by the idea of direct manipulation [56, 57].”
Wu, T., Terry, M., & Cai, C. J. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517582 · Cited by 483 · cited 2×
“For instance, few-shot prompting [6, 19, 79] and chain-of-thought prompting [68, 72, 74] are both representative prompting techniques for generative language models.”
BackgroundMethodCogView: Mastering Text-to-Image Generation via Transformers
Ding, M., Yang, Z., Hong, W., Zheng, W., Zhou, C., Yin, D., Lin, J., Zou, X., Shao, Z., Yang, H., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.13290 · Cited by 450
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436 · cited 1×
“We found that participants felt their generated images were more aesthetically pleasing (median rating: 6) and closer to their expectations (median rating: 6) when using PromptCharm compared with using either Baseline (median ratings: 5 and 4.5) or Promp-tist (median ratings: 5 and 5).”
MethodInfluentialHuman-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making
Cai, C. J., Reif, E., Hegde, N., Hipp, J., Kim, B., Smilkov, D., Wattenberg, M., Viegas, F., Corrado, G. S., Stumpe, M. C., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300234 · Cited by 405 · cited 1×
“As a result, the design of PromptCharm is highly motivated by the recent guidelines and design principles for human-AI collaboration [3, 7, 8, 14, 34, 61].”
NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion
Wu, C., Liang, J., Ji, L., Yang, F., Fang, Y., Jiang, D., & Duan, N. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19787-1_41 · Cited by 398
Prompt-to-Prompt Image Editing with Cross Attention Control
Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., & Cohen‐Or, D. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.01626 · Cited by 360 · cited 1×
“In the end, we discuss the design implications from PromptCharm and propose several interesting future research opportunities.”
BackgroundA Review of User Interface Design for Interactive Machine Learning
Dudley, J. J., & Kristensson, P. O. (2018) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3185517 · Cited by 343 · cited 1×
“As a result, the design of PromptCharm is highly motivated by the recent guidelines and design principles for human-AI collaboration [3, 7, 8, 14, 34, 61].”
Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI
Wang, D., Weisz, J. D., Muller, M., Ram, P., Geyer, W., Dugan, C., Tausczik, Y., Samulowitz, H., & Gray, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359313 · Cited by 342 · cited 1×
“As a result, the design of PromptCharm is highly motivated by the recent guidelines and design principles for human-AI collaboration [3, 7, 8, 14, 34, 61].”
MethodLAION-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1833 · Cited by 325
Novice-AI Music Co-Creation via AI-Steering Tools for Deep Generative Models
Louie, R., Coenen, A., Huang, C. Z., Terry, M., & Cai, C. J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376739 · Cited by 271 · cited 1×
“These results indicate that the effectiveness of PromptCharm is not strongly correlated to user expertise.”
Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models
Strobelt, H., Webson, A., Sanh, V., Hoover, B., Beyer, J., Pfister, H., & Rush, A. M. (2022) · arXiv · doi:10.48550/arxiv.2208.07852 · Cited by 204
Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation With Large Language Models
Strobelt, H., Webson, A., Sanh, V., Hoover, B., Beyer, J., Pfister, H., & Rush, A. M. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209479 · Cited by 204
KVT: k-NN Attention for Boosting Vision Transformers
Wang, P., Wang, X., Wang, F., Lin, M., Chang, S., Li, H., & Jin, R. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20053-3_17 · Cited by 199
Brade, S., Wang, B., Sousa, M., Oore, S., & Grossman, T. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606725 · Cited by 198
Large-scale Text-to-Image Generation Models for Visual Artists’ Creative Works
Ko, H.-K., Park, G., Jeon, H., Jo, J., Kim, J., & Seo, J. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584078 · Cited by 189 · cited 7×
“As a complementary study to the prior research, Ko et al. conducted an interview study with 28 visual artists to help the research community understand the potential and design guidelines of using large-scale text-to-image models for visual art creations [31].”
BackgroundMethodResultInfluentialDiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models
Wang, Z. J., Montoya, E., Munechika, D., Yang, H., Hoover, B., & Chau, D. H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.51 · Cited by 171
Self-citation3DALL-E: Integrating Text-to-Image AI in 3D Design Workflows
Liu, V., Vermeulen, J., Fitzmaurice, G., & Matejka, J. (2023) · Proceedings of the 2023 ACM Designing Interactive Systems Conference · doi:10.1145/3563657.3596098 · Cited by 163 · cited 3×
“3DALL-E integrates OpenAI’s DALL-E (a text-to-image generative model), GPT-3 and CLIP to inspire professional CAD designers’ 3D design work through 2D images’ prototyping [38].”
BackgroundMethodSemantic shape editing using deformation handles
Yumer, M. E., Chaudhuri, S., Hodgins, J. K., & Kara, L. B. (2015) · ACM Transactions on Graphics · doi:10.1145/2766908 · Cited by 145 · cited 1×
“Early attempts on generative design covering both 2D [10, 76] and 3D design [9, 10, 29, 42, 75], which mostly focus on assisting users in exploring a diverse set of design alternatives.”
Dream Lens: Exploration and Visualization of Large-Scale Generative Design Datasets
Matejka, J., Glueck, M., Bradner, E., Hashemi, A., Grossman, T., & Fitzmaurice, G. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173943 · Cited by 144 · cited 2×
“For instance, Matejka et al. proposed DreamLens, an interactive system for exploring and visualizing large-scale generative design datasets [43].”
BackgroundMethodDreamSketch: Early Stage 3D Design Explorations with Sketching and Generative Design
Kazi, R. H., Grossman, T., Cheong, H., Hashemi, A., & Fitzmaurice, G. (2017) · Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3126594.3126662 · Cited by 142 · cited 1×
“Early attempts on generative design covering both 2D [10, 76] and 3D design [9, 10, 29, 42, 75], which mostly focus on assisting users in exploring a diverse set of design alternatives.”
BackgroundPromptMagician: Interactive Prompt Engineering for Text-to-Image Creation
Feng, Y., Wang, X., Wong, K. K., Wang, S., Lu, Y., Zhu, M., Wang, B., & Chen, W. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3327168 · Cited by 124 · cited 6×
“These approaches aim to guide the iterative prompt refinement process by rendering a set of alternative prompts [58] or suggesting a few new keywords for users to choose from [5, 18].”
BackgroundInfluentialMaking Pre-trained Language Models Better Few-shot Learners
Gao, T., Fisch, A., & Chen, D. (2020) · arXiv · doi:10.48550/arxiv.2012.15723 · Cited by 124 · cited 1×
“For instance, few-shot prompting [6, 19, 79] and chain-of-thought prompting [68, 72, 74] are both representative prompting techniques for generative language models.”
BackgroundAttribit: content creation with semantic attributes
Chaudhuri, S., Kalogerakis, E., Giguere, S., & Funkhouser, T. (2013) · Proceedings of the 26th annual ACM symposium on User interface software and technology · doi:10.1145/2501988.2502008 · Cited by 123 · cited 1×
“Early attempts on generative design covering both 2D [10, 76] and 3D design [9, 10, 29, 42, 75], which mostly focus on assisting users in exploring a diverse set of design alternatives.”
BackgroundRePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressions
Wang, Y., Shen, S., & Lim, B. Y. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581402 · Cited by 120 · cited 3×
“Since hand-crafting prompts require significant manual efforts, another line of research has focused on automated prompt generation [24, 46, 55, 65, 70].”
BackgroundMethodWhat the DAAM: Interpreting Stable Diffusion Using Cross Attention
Tang, R., Liu, L., Pandey, A., Jiang, Z., Yang, G., Kumar, K., Stenetorp, P., Lin, J., & Ture, F. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.310 · Cited by 118
Opal: Multimodal Image Generation for News Illustration
Liu, V., Qiao, H., & Chilton, L. (2022) · Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3526113.3545621 · Cited by 104 · cited 5×
“Though we have only evaluated PromptCharm on the Stable Diffusion , we believe the design of PromptCharm can generalize to other open-source text-to-image generation models, e.g., CogView [13], VQGAN+CLIP 11 .”
BackgroundMethodInfluentialPromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions
Chung, J. J. Y., & Adar, E. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606777 · Cited by 96 · cited 1×
“Another recent work, PromptPaint, also supports prompting beyond text by providing flexible steering through paint medium-like interactions [12].”
BackgroundOn the Design of AI-powered Code Assistants for Notebooks
Mcnutt, A. M., Wang, C., Deline, R. A., & Drucker, S. M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580940 · Cited by 77 · cited 1×
“We found that participants felt their generated images were more aesthetically pleasing (median rating: 6) and closer to their expectations (median rating: 6) when using PromptCharm compared with using either Baseline (median ratings: 5 and 4.5) or Promp-tist (median ratings: 5 and 5).”
MethodInfluentialHard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery
Wen, Y., Jain, N., Kirchenbauer, J., Goldblum, M., Geiping, J., & Goldstein, T. (2023) · arXiv · doi:10.48550/arxiv.2302.03668 · Cited by 77 · cited 3×
“Since hand-crafting prompts require significant manual efforts, another line of research has focused on automated prompt generation [24, 46, 55, 65, 70].”
BackgroundMethodBest Prompts for Text-to-Image Models and How to Find Them
Pavlichenko, N., & Ustalov, D. (2023) · Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3539618.3592000 · Cited by 71
PopBlends: Strategies for Conceptual Blending with Large Language Models
Wang, S., Petridis, S., Kwon, T., Ma, X., & Chilton, L. B. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580948 · Cited by 64
GEM-NI: A System for Creating and Managing Alternatives In Generative Design
Zaman, L., Stuerzlinger, W., Neugebauer, C., Woodbury, R., Elkhaldi, M., Shireen, N., & Terry, M. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702398 · Cited by 59 · cited 1×
“Early attempts on generative design covering both 2D [10, 76] and 3D design [9, 10, 29, 42, 75], which mostly focus on assisting users in exploring a diverse set of design alternatives.”
BackgroundForte: User-Driven Generative Design
Chen, X. '., Tao, Y., Wang, G., Kang, R., Grossman, T., Coros, S., & Hudson, S. E. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174070 · Cited by 59 · cited 1×
“Early attempts on generative design covering both 2D [10, 76] and 3D design [9, 10, 29, 42, 75], which mostly focus on assisting users in exploring a diverse set of design alternatives.”
BackgroundGenerating Images from Captions with Attention
Mansimov, E., Parisotto, E., Ba, J. L., & Salakhutdinov, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.02793 · Cited by 59 · cited 2×
“AlignDraw is extended from Draw [22], an RNN-based image generation model, by leveraging a bidirectional attention RNN language model to guide the image generation process.”
BackgroundToward General Design Principles for Generative AI Applications
Weisz, J. D., Muller, M., He, J., & Houde, S. (2023) · arXiv · doi:10.48550/arxiv.2301.05578 · Cited by 54
FlatMagic: Improving Flat Colorization through AI-driven Design for Digital Comic Professionals
Yan, C., Chung, J. J. Y., Kiheon, Y., Gingold, Y., Adar, E., & Hong, S. R. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502075 · Cited by 50 · cited 2×
“The most related generative design work to us includes those using AI models to support user’s design [11, 17, 31, 37, 38, 64, 73].”
MethodEcco: An Open Source Library for the Explainability of Transformer Language Models
Alammar, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations · doi:10.18653/v1/2021.acl-demo.30 · Cited by 44 · cited 1×
“The refined prompt will show up in the middle text”
BackgroundPrompt Engineering for Text-Based Generative Art
Oppenlaender, J. (2022) · arXiv.org · doi:10.48550/arxiv.2204.13988 · Cited by 44
Uncurated Image-Text Datasets: Shedding Light on Demographic Bias
Garcia, N., Hirota, Y., Wu, Y., & Nakashima, Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00672 · Cited by 43 · cited 1×
“Moreover, recent studies from the ML community have shown that SOTA generative AI and large language models could actually misinterpret the user’s intention in the input text prompt [20, 32].”
BackgroundShneiderman, B. (1981) · doi:10.1145/800276.810991 · Cited by 39
Optimizing Prompts for Text-to-Image GenerationPreprint
Hao, Y., Chi, Z., Li, D., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.09611 · Cited by 38
VisiFit: Structuring Iterative Improvement for Novice Designers
Chilton, L. B., Ozmen, E. J., Ross, S. H., & Liu, V. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445089 · Cited by 33 · cited 2×
“The most related generative design work to us includes those using AI models to support user’s design [11, 17, 31, 37, 38, 64, 73].”
MethodGrounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2022) · arXiv · doi:10.48550/arxiv.2206.15000 · Cited by 23 · cited 1×
“We found that participants felt their generated images were more aesthetically pleasing (median rating: 6) and closer to their expectations (median rating: 6) when using PromptCharm compared with using either Baseline (median ratings: 5 and 4.5) or Promp-tist (median ratings: 5 and 5).”
MethodInfluentialGANzilla: User-Driven Direction Discovery in Generative Adversarial Networks
Evirgen, N., & Chen, X. '. (2022) · Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3526113.3545638 · Cited by 22
GANravel: User-Driven Direction Disentanglement in Generative Adversarial Networks
Evirgen, N., & Chen, X. (2023) · International Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581226 · Cited by 21
Kou, B., Chen, S.-M., Wang, Z.-J., Ma, L., & Zhang, T.-Y. (2023) · arXiv.org · doi:10.48550/arxiv.2306.01220 · Cited by 20 · cited 1×
“Moreover, recent studies from the ML community have shown that SOTA generative AI and large language models could actually misinterpret the user’s intention in the input text prompt [20, 32].”
Self-citationEfficient Human-in-the-loop System for Guiding DNNs Attention
He, Y., Yang, X., Chang, C.-M., Xie, H., & Igarashi, T. (2023) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3581641.3584074 · Cited by 11
Research works printed but not linked: 12
- 10.18653/v1 · doi:10.18653/v1
- [21] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Advances in Neural Information Processing Systems , Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (Eds.), Vol. 27. Curran Associates, Inc. -
- [22] Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Rezende, and Daan Wierstra. 2015. DRAW: A Recurrent Neural Network For Image Generation. In Proceedings of the 32nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 37) , Francis Bach and David Blei (Eds.). PMLR, Lille, France, 1462-1471. -
- [25] Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in psychology . Vol. 52. Elsevier, 139-183. -
- [30] Amy J Ko, Brad A Myers, and Htet Htet Aung. 2004. Six Learning Barriers in EndUser Programming Systems. In 2004 IEEE Symposium on Visual Languages-Human Centric Computing . IEEE, 199-206. -
- [33] Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip Torr. 2019. Controllable Text-to-Image Generation. In Advances in Neural Information Processing Systems , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc. -
- [40] Scott M Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. In Advances in Neural Information Processing Systems , I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc. https://proceedings.neurips.cc/paper\_ files/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf -
- [50] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-Shot Text-to-Image Generation. In Proceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 139) , Marina Meila and Tong Zhang (Eds.). PMLR, 8821-8831. -
- 10.18653/v1 · doi:10.18653/v1
- [60] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems , I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc. -
- [63] Qing Wang, Jiaming Zhang, Kailun Yang, Kunyu Peng, and Rainer Stiefelhagen. 2022. MatchFormer: Interleaving Attention in Transformers for Feature Matching. In Proceedings of the Asian Conference on Computer Vision (ACCV) . 2746-2762. -
- [79] Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate Before Use: Improving Few-Shot Performance of Language Models. In Proceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 139) , Marina Meila and Tong Zhang (Eds.). PMLR, 1269712706.
Cited by
AIdeation: Designing a Human-AI Collaborative Ideation System for Concept Designers
Wang, W.-F., Lu, C.-T., Ponsa i Campanyà, N., Chen, B.-Y., & Chen, M. Y. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714148 · Cited by 50
What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use
Ma, Q., Peng, W., Yang, C., Shen, H., Koedinger, K. R., & Wu, T. (2025) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3731756 · Cited by 113
Designing Interactions with Generative AI for Art and Creativity: A Systematic Review and Taxonomy
Hu, X., Xing, Y., Cai, X., Zhao, Y., Cook, M., Borgo, R., & Neate, T. (2025) · Designing Interactive Systems Conference · doi:10.1145/3715336.3735843 · Cited by 61
Zhang, S., Wang, H., & Yi, X. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757594 · Cited by 36
Riche, N. H., Offenwanger, A., Gmeiner, F., Brown, D., Romat, H., Pahud, M., Marquardt, N., Inkpen, K. M., & Hinckley, K. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714259 · Cited by 62
Metadata
- Authors
- Wang, Z., Huang, Y., Song, D., Ma, L., & Zhang, T.
- Affiliations
- University of Alberta (CA)
- Purdue University West Lafayette (US)
- Venue
- Proceedings of the CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2024-05-11
- Pages
- 1-21
- OpenAlex
- W4392619076
- Semantic Scholar
- c6e2837894bb545fc62ef41d7dadcd6e1bb51dc5
- Cited by
- 115
- References
- 55
- Citations by year
- 2026: 41 · 2025: 66 · 2024: 8
- FWCI
- 23.82
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Natural Sciences and Engineering Research Council of Canada (No.RGPIN-2021-02549, No.RGPAS-2021-00034, No.DGECR-2021-00019)
- JST-Mirai Program Grant (No.JPMJMI20B8)
- JSPS KAKENHI Grant (No.JP21H04877, No.JP23H03372)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Digital Media and Visual Art
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Computer Graphics and Visualization Techniques
- Keywords
- text-to-image generation, prompt engineering, prompt refinement, mixed-initiative systems, Stable Diffusion, attention visualization, image inpainting, novice users
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.