Initial Images: Using Image Prompts to Improve Subject Representation in Multimodal AI Generated Art
Abstract
Advances in text-to-image generative models have made it easier for people to create art by just prompting models with text. However, creating through text leaves users with limited control over the final composition or the way the subject is represented. A potential solution is to use image prompts alongside text prompts to condition the model. To better understand how and when image prompts can improve subject representation in generations, we conduct an annotation experiment to quantify their effect on generations of abstract, concrete plural, and concrete singular subjects. We find that initial images improved subject representation across all subject types, with the most noticeable improvement in concrete singular subjects. In an analysis of different types of initial images, we find that icons and photos produced high quality generations of different aesthetics. We conclude with design guidelines for how initial images can improve subject representation in AI art.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Transformer models have significantly revolutionized AI, especially NLP, due to their higher efficiency and scalability.
“Transformer models have significantly revolutionized the field of AI, especially in NLP, due to their higher efficiency and scalability [139], [140], [141].”
From From Google Gemini to OpenAI Q* (Q-Star): A Survey on Reshaping the Generative Artificial Intelligence (AI) Research Landscape (2025) · cites this paper as [140]
Cites
Analyzing and Improving the Image Quality of StyleGAN
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., & Aila, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00813 · Cited by 6,332 · see citing works
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery
Patashnik, O., Wu, Z., Shechtman, E., Cohen‐Or, D., & Lischinski, D. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00209 · Cited by 1,079 · see citing works
Neuroanatomical correlates of aesthetic preference for paintings
Vartanian, O. A., & Goel, V. (2004) · Neuroreport · doi:10.1097/00001756-200404090-00032 · Cited by 470 · see citing works
Cupchik, G. C., Vartanian, O. A., Crawley, A. P., & Mikulis, D. J. (2009) · Brain and Cognition · doi:10.1016/j.bandc.2009.01.003 · Cited by 422 · see citing works
Perceptual and Physiological Responses to Jackson Pollock's Fractals
Taylor, R. P., Špehar, B., van Donkelaar, P., & Hagerhall, C. M. P. (2011) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2011.00060 · Cited by 200 · see citing works
Visual interest in pictorial art during an aesthetic experience
Nodine, C. F., Mello-Thoms, C. R., Krupinski, E. A., & Locher, P. J. (2007) · Spatial Vision · doi:10.1163/156856807782753868 · Cited by 152 · see citing works
Dream Lens: Exploration and Visualization of Large-Scale Generative Design Datasets
Matejka, J., Glueck, M., Bradner, E., Hashemi, A., Grossman, T., & Fitzmaurice, G. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173943 · Cited by 144 · see citing works
Interactive Sketch & Fill: Multiclass Sketch-to-Image Translation
Ghosh, A., Zhang, R., Dokania, P. K., Wang, O., Efros, A. A., Torr, P. H. S., & Shechtman, E. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00126 · Cited by 130 · see citing works
CLIPDraw: Exploring Text-To-Drawing Synthesis Through Language-Image Encoders
Frans, K., Soros, L. B., & Witkowski, O. (2022) · neural information processing systems · doi:10.52202/068431-0376 · Cited by 127 · see citing works
Chaudhuri, S., Kalogerakis, E., Giguere, S., & Funkhouser, T. A. (2013) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2501988.2502008 · Cited by 123 · see citing works
Style follows content: On the microgenesis of art perception
Augustin, M., Leder, H., Hutzler, F., & Carbon, C. (2008) · Acta Psychologica · doi:10.1016/j.actpsy.2007.11.006 · Cited by 101 · see citing works
CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image EncodersPreprint
Frans, K., Soros, L. B., & Witkowski, O. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.14843 · Cited by 92 · see citing works
Reductionism in Art and Brain Science: Bridging the Two Cultures
Ione, A. (2017) · Leonardo · doi:10.1162/leon_r_01438 · Cited by 55 · see citing works
What does the brain tell us about abstract art?
Aviv, V. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2014.00085 · Cited by 41 · see citing works
Evaluating Creativity in Computational Co-Creative SystemsPreprint
Karimi, P., Grace, K. S., Maher, M. L., & Davis, N. M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.09886 · Cited by 34 · see citing works
Shimizu, E., Fisher, M., Paris, S., McCann, J., & Fatahalian, K. (2020) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3379337.3415866 · Cited by 23 · see citing works
Nake, F. (2007) · doi:10.1145/1254960.1255041 · Cited by 6 · see citing works
crea.blender: A Neural Network-Based Image Generation Game to Assess Creativity
Rafner, J. F., Hjorth, A., Risi, S., Philipsen, L., Dumas, C., Biskjær, M. M., Noy, L., Tylén, K., Bergenholtz, C., Lynch, J., et al. (2020) · IT University Of Copenhagen (IT University of Copenhagen) · doi:10.48550/arxiv.2008.05914 · Cited by 5 · see citing works
Anadol, R. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3355700 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Generative AI and ChatGPT: applications, challenges, and AI-human collaboration
Fui-Hoon Nah, F., Zheng, R., Cai, J., Siau, K., & Chen, L. (2023) · Journal of Information Technology Case and Application Research · doi:10.1080/15228053.2023.2233814 · Cited by 1,363 · see citing works
Generative AI Literacy: Twelve Defining Competencies
Annapureddy, R., Fornaroli, A., & Gatica-Perez, D. (2025) · Digital Government: Research and Practice · doi:10.1145/3685680 · Cited by 333 · see citing works
Liu, M., Zhang, L. J., & Biebricher, C. (2023) · Computers & Education · doi:10.1016/j.compedu.2023.104977 · Cited by 372 · see citing works
A taxonomy of prompt modifiers for text-to-image generation
Oppenlaender, J. (2023) · Behaviour and Information Technology · doi:10.1080/0144929x.2023.2286532 · Cited by 321 · see citing works
Prompting AI Art: An Investigation into the Creative Skill of Prompt Engineering
Oppenlaender, J., Linder, R., & Silvennoinen, J. M. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2431761 · Cited by 308 · see citing works
Large-scale Text-to-Image Generation Models for Visual Artists’ Creative Works
Ko, H.-K., Park, G., Jeon, H., Jo, J., Kim, J., & Seo, J. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584078 · Cited by 189 · see citing works
DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models
Wang, Z. J., Montoya, E., Munechika, D., Yang, H., Hoover, B., & Chau, D. H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.51 · Cited by 244 · see citing works
3DALL-E: Integrating Text-to-Image AI in 3D Design Workflows
Liu, V., Vermeulen, J., Fitzmaurice, G. W., & Matejka, J. (2023) · Designing Interactive Systems Conference · doi:10.1145/3563657.3596098 · Cited by 179 · see citing works
Self-citationRePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressions
Wang, Y., Shen, S., & Lim, B. Y. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581402 · Cited by 120 · see citing works
McIntosh, T. R., Susnjak, T., Liu, T., Watters, P., Xu, D., Liu, D., & Halgamuge, M. N. (2025) · Technologies · doi:10.3390/technologies13020051 · Cited by 102 · see citing works
“Advanced algorithms and multimodal attention mechanisms are needed to balance attention across different input media and resolve conflicts between modalities, especially when they provide contradictory information [119], [120], [118].”
BackgroundID.8: Co-Creating Visual Stories with Generative AI
Antony, V. N., & Huang, C.-M. (2024) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3672277 · Cited by 81 · see citing works
He, Z., Su, J., Chen, L., Wang, T., & Lc, R. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3711006 · Cited by 67 · see citing works
Huang, R., Lin, H., Chen, C., Kang Zhang, A., & Zeng, W. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642824 · Cited by 51 · see citing works
Beyond Text-to-Image: Multimodal Prompts to Explore Generative AI
Liu, V. (2023) · Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544549.3577043 · Cited by 38 · see citing works
Self-citationPark, H., Eirich, J., Luckow, A., & Sedlmair, M. (2024) · Nordic Conference on Human-Computer Interaction · doi:10.1145/3679318.3685370 · Cited by 30 · see citing works
Diffusion-Based Visual Art Creation: A Survey and New Perspectives
Wang, B., Chen, Q., & Wang, Z. (2025) · ACM Computing Surveys · doi:10.1145/3728459 · Cited by 40 · see citing works
Hussain, M. (2023) · IEEE Access · doi:10.1109/access.2023.3332468 · Cited by 33 · see citing works
GenFrame – Embedding Generative AI Into Interactive Artifacts
Kun, P., Freiberger, M., Løvlie, A. S., & Risi, S. (2024) · Designing Interactive Systems Conference · doi:10.1145/3643834.3660750 · Cited by 15 · see citing works
Cultural Relevance Index: Measuring Cultural Relevance in AI-Generated Images
Elsharif, W., Agus, M., Alzubaidi, M., & She, J. P. M. (2024) · IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR) · doi:10.1109/mipr62202.2024.00071 · Cited by 9 · see citing works
Zhang, X., Zhu, S., Zhao, Y. C., Jia, M., & Zhu, Q. (2024) · Information Research · doi:10.47989/ir292856 · Cited by 8 · see citing works
Public Service with Generative AI: Exploring Features and Applications
Aryfiyanto, H., & Alamsyah, A. (2024) · International Conference of Computer and Informatics Engineering (IC2IE) · doi:10.1109/ic2ie63342.2024.10747963 · Cited by 11 · see citing works
Zheng, C., Yu, Z., Jiang, Y., Zhang, M., Lu, X., Jin, J., & Gao, L. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713274 · Cited by 7 · see citing works
Bir Diyalog Ortamı Olarak Üretken Yapay Zeka: Tasarımda Anlamsal Arayış Sürecinin Temsili
Ozar, B., & Koca, D. (2024) · Art-e Sanat Dergisi · doi:10.21602/sduarte.1462578 · Cited by 7 · see citing works
Understanding User Perceptions and the Role of AI Image Generators in Image Creation Workflows
Han, S.-J., & Fussell, S. R. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713227 · Cited by 5 · see citing works
MVPrompt: Building Music-Visual Prompts for AI Artists to Craft Music Video Mise-en-scène
Lee, C., Lee, D., & Hong, J.-H. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713876 · Cited by 3 · see citing works
Koh, J. T. K. V. (2023) · Springer series on cultural computing · doi:10.1007/978-3-031-28138-9_6 · Cited by 3 · see citing works
When Art Meets Algorithm: Exploring How People Perceive Meaning in Human–AI Collaborative Art
Döring, T., Chae, R. L., Efendić, E., Finken, D., & Nikulina, O. (2026) · Academy of Management Discoveries · doi:10.5465/amd.2024.0285 · Cited by 2 · see citing works
Li, J., Wang, X., Liu, C., Chen, Y., Fang, L., Xu, Y., Zhang, L., Lee, K.-P., & Wang, S. J. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791643 · Cited by 1 · see citing works
The Generative Fairy Tale of Scary Little Red Riding Hood
Harde, L., Jensen, L. G., von Krogh, J., Plesner, A., Sørensen, O., & Pohl, H. (2024) · ACM International Conference on Interactive Media Experiences · doi:10.1145/3639701.3656303 · Cited by 1 · see citing works
Full-Process Structural Prompts: Enhancing the Expressive Effects of GAI in Idiom Image Generation
Tang, R., & Jingzhou, W. (2024) · International Conference on Control and Intelligent Robotics (ICCIR) · doi:10.1145/3687488.3687548 · Cited by 1 · see citing works
Crafting AI-Driven Open Educational Resources (OER) Videos for Enhanced German Language Learning
Kharis, M. (2024) · IEEE International Symposium on Consumer Electronics (ISCE) · doi:10.1109/isct62336.2024.10791276 · Cited by 1 · see citing works
Crafting Seeding Imagery for Artmaking Generative AI Tools
Hai-Jew, S. (2024) · Advances in multimedia and interactive technologies book series · doi:10.4018/979-8-3693-2927-6.ch002
Multimodal Outputs for the Workplace From Generative AI
Hai-Jew, S. (2024) · Advances in multimedia and interactive technologies book series · doi:10.4018/979-8-3693-2927-6.ch008
Hai-Jew, S. (2024) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-3382-2.ch009
PMAT: An Approach of Multimodal Prompting for Robot Assembly Task
Yu, D., Fan, X., Chen, H., Li, Y., Zheng, D., & Li, H. (2024) · IEEE International Conference on Cyber Technology in Automation, Control, and Intelligent Systems · doi:10.1109/cyber63482.2024.10749643
A Dilemma-Based Learning-to-Rank Approach for Generative Design in Urban Architectural Regeneration
Kavouras, I., Rallis, I., Zouli, D., Sardis, E., Protopapadakis, E. E., Doulamis, A. D., & Doulamis, N. D. (2024) · Algorithms · doi:10.3390/a17120538
AI-enhanced imagery rehearsal therapy: A novel approach to treating nightmare disorder
Kahn, M. (2024) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2024.102046
Generative AI in the Co-creation of Abstract Images
Amorim, M. F., & da Silva, B. M. (2024) · Springer series in design and innovation · doi:10.1007/978-3-031-77566-6_69
Fine-Tuned Large Language Model for Banking
Limkar, S. V., Nale, S., Darade, S., Chaudhari, A., & Taras, A. (2025) · Smart innovation, systems and technologies · doi:10.1007/978-981-96-0139-4_35
Evaluating Visual Prompting Modalities for Generative AI-Assisted UI Design
Calò, T., & Russis, L. D. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-95452-8_11
Edén, A. S., Faraon, M., & Rönkkö, K. (2026) · International Conference on Tangible, Embedded, and Embodied Interaction (TEI) · doi:10.1145/3731459.3773314
From thought to visual composition: a brain-driven visual blends technique for visual blending tasks
Zhang, X., Kleinsmann, M. S., Wang, S. J., Yan, D., Wei, Z., & Wang, P. (2026) · Journal of Engineering Design · doi:10.1080/09544828.2026.2639935
Lim, K. Y. T., Nguyen, D. M. A., & Thien, M. T. N. (2026) · Springer eBooks · doi:10.1007/978-3-032-12174-5_26 · Cited by 2 · see citing works
Beyond Prompts: Reference-Based Interactions for Generative AI in Design Visualization
Sanatani, R. P., Gupta, R., Tan, F. H., Nagakura, T., & Davis, R. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-30846-7_10
- Authors
- Qiao, H., Liu, V., & Chilton, L.
- Affiliations
- Columbia University (US)
- Venue
- Creativity and Cognition
- Publisher
- ACM
- Published
- 2022-06-20
- Pages
- 15-28
- OpenAlex
- W4283167234
- Cited by
- 46
- References
- 18
- Citations by year
- 2026: 6 · 2025: 8 · 2024: 22 · 2023: 10
- FWCI
- 3.63
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Creativity and Cognition (publishedVersion)
- Funders
- NSF (National Science Foundation) (1644869)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Digital Media and Visual Art
- Social Sciences › Arts and Humanities › Visual Arts and Performing Arts › Art, Technology, and Culture
- Keywords
- image prompts, text-to-image generation, multimodal conditioning, subject representation, AI-generated art, icons, aesthetic preference, design guidelines
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
