CoChat
Sign Up
Cited by 7Generative Adversarial Networks and Image SynthesisMultimodal Machine Learning ApplicationsRecommender Systems and Techniques

The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.

Capability-aware Prompt Reformulation Learning for Text-to-Image Generation

Open access
Zhan, Jingtao, Ai, Qingyao, Liu, Yiqun, Chen, Jia, Ma, Shaoping·2024·Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval
DOI 10.1145/3626772.3657787
Open this paper in CoChat — free account

Abstract

Text-to-image generation systems have emerged as revolutionary tools in the realm of artistic creation, offering unprecedented ease in transforming textual prompts into visual art. However, the efficacy of these systems is intricately linked to the quality of user-provided prompts, which often poses a challenge to users unfamiliar with prompt crafting. This paper addresses this challenge by leveraging user reformulation data from interaction logs to develop an automatic prompt reformulation model. Our in-depth analysis of these logs reveals that user prompt reformulation is heavily dependent on the individual user's capability, resulting in significant variance in the quality of reformulation pairs. To effectively use this data for training, we introduce the Capability-aware Prompt Reformulation (CAPR) framework. CAPR innovatively integrates user capability into the reformulation process through two key components: the Conditional Reformulation Model (CRM) and Configurable Capability Features (CCF). CRM reformulates prompts according to a specified user capability, as represented by CCF. The CCF, in turn, offers the flexibility to tune and guide the CRM's behavior. This enables CAPR to effectively learn diverse reformulation strategies across various user capacities and to simulate high-capability user reformulation during inference. Extensive experiments demonstrate CAPR's superior performance over existing baselines and its remarkable robustness on unseen systems. Furthermore, comprehensive analyses validate the effectiveness of different components. CAPR can facilitate user-friendly interaction with text-to-image systems and make advanced artistic creation more achievable for a broader range of users.

Overview

Integrity signals

Registry

Crossref · 10.1145/3626772.3657787

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • High-quality prompt writers can be automatically learned based on downstream task performance and user logs in image generation.

    “Existing studies have shown that high-quality prompt writers can be automatically learned based on downstream task performance and user logs in image generation [53]”

    From Foundations of GenIR (2025) · cites this paper as [53]

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 44

  1. 10.1007/978-3-540-30192-9_58 · DOI
  2. Paolo Boldi, Francesco Bonchi, Carlos Castillo, and Sebastiano Vigna. 2011. Query reformulation mining: models, patterns, and applications. Information retrieval 14 (2011), 257--289.
  3. 10.1145/3586183.3606725 · DOI · Paper page
  4. Tom Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared D Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020) 1877--1901.
  5. Fei Cai, Maarten De Rijke, et al. 2016. A survey of query auto completion in information retrieval. Foundations and Trends® in Information Retrieval 10, 4 (2016), 273--363.
  6. 10.1145/2071389.2071390 · DOI · Paper page
  7. 10.1002/asi.20210 · DOI
  8. 10.1145/3448127 · DOI
  9. 10.1145/3442381.3450127 · DOI
  10. 10.1016/j.ipm.2018.06.009 · DOI
  11. Jerry Zikun Chen, Shi Yu, and Haoran Wang. 2020. Exploring Fluent Query Reformulations with Text-to-Text Transformers and Reinforcement Learning. arXiv preprint arXiv:2012.10033 (2020). · arXiv
  12. Niklas Deckers, Julia Peters, and Martin Potthast. 2023. Manipulating Embeddings of Stable Diffusion Prompts. arXiv preprint arXiv:2308.12059 (2023). · arXiv
  13. 10.1145/3132847.3133010 · DOI
  14. 10.5555/351325.351327 · DOI
  15. Archan Ghosh, Debgandhar Ghosh, Madhurima Maji, Suchinta Chanda, and Kalporup Goswami. 2023. MTTN: Multi-Pair Text to Text Narratives for Prompt Generation. arXiv preprint arXiv:2301.10172 (2023). · arXiv
  16. 10.1145/2484028.2484055 · DOI
  17. Kishaloy Halder, Heng-Tze Cheng, Ellie Ka In Chio, Georgios Roumpos, Tao Wu, and Ritesh Agarwal. 2020. Modeling Information Need of Users in Search Sessions. arXiv preprint arXiv:2001.00861 (2020). · arXiv
  18. Yaru Hao, Zewen Chi, Li Dong, and Furu Wei. 2022. Optimizing prompts for text-to-image generation. arXiv preprint arXiv:2212.09611 (2022). · arXiv
  19. Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. Advances in neural information processing systems 33 (2020), 6840--6851.
  20. 10.1145/2600428.2609614 · DOI
  21. 10.1109/CVPR52729.2023.02239 · DOI
  22. 10.1109/CVPR52729.2023.00976 · DOI
  23. Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. 2021. Variational diffusion models. Advances in neural information processing systems 34 (2021), 21696--21707.
  24. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014). · arXiv
  25. 10.1109/CVPR52688.2022.01765 · DOI
  26. 10.1109/CVPR52729.2023.01371 · DOI
  27. Midjourney. 2023. Midjourney: An Independent Research Lab Exploring New Mediums of Thought. https://www.midjourney.com/. [Online; accessed 21- January-2024].
  28. Jonas Oppenlaender. 2022. A taxonomy of prompt modifiers for text-to-image generation. arXiv preprint arXiv:2204.13988 2 (2022). · arXiv
  29. Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. 2023. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. arXiv:2307.01952 [cs.CV] · arXiv
  30. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 8748--8763.
  31. 10.1002/(SICI)1097-4571(199006)41:4<288::AID-ASI8>3.0.CO;2-H · DOI
  32. Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. 2022. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems 35 (2022), 25278--25294.
  33. Xinyue Shen, Yiting Qu, Michael Backes, and Yang Zhang. 2023. Prompt Stealing Attacks Against Text-to-Image Generation Models. arXiv preprint arXiv:2302.09923 (2023). · arXiv
  34. Craig Silverstein, Hannes Marais, Monika Henzinger, and Michael Moricz. 1999. Analysis of a very large web search engine query log. In Acm sigir forum, Vol. 33. ACM New York, NY, USA, 6--12.
  35. 10.1145/2806416.2806493 · DOI
  36. Amanda Spink, Bernard J Jansen, and H Cenk Ozmultu. 2000. Use of query reformulation and relevance feedback by Excite users. Internet research 10, 4 (2000), 317--328.
  37. 10.18653/v1/2023.acl-long.51 · DOI
  38. Sam Witteveen and Martin Andrews. 2022. Investigating prompt engineering in diffusion models. arXiv preprint arXiv:2211.15462 (2022). · arXiv
  39. Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. 2023. Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis. arXiv preprint arXiv:2306.09341 (2023). · arXiv
  40. 10.1145/3543507.3587430 · DOI
  41. Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. 2023. Imagereward: Learning and evaluating human preferences for text-to-image generation. arXiv preprint arXiv:2304.05977 (2023). · arXiv
  42. Keyulu Xu, Mozhi Zhang, Jingling Li, Simon S Du, Ken-ichi Kawarabayashi, and Stefanie Jegelka. 2020. How neural networks extrapolate: From feedforward to graph neural networks. arXiv preprint arXiv:2009.11848 (2020). · arXiv
  43. Peiyuan Zhang Guangtao Zeng Tianduo Wang and Wei Lu. 2024. TinyLlama: An Open-Source Small Language Model. arXiv:2401.02385 [cs.CL] · arXiv
  44. 10.1109/CVPR52729.2023.00979 · DOI

Cited by

Sorted by citations

Metadata

Authors
Zhan, J., Ai, Q., Liu, Y., Chen, J., & Ma, S.
Affiliations
  • Quan Cheng Laboratory (CN)
  • Tsinghua University (CN)
  • Xiaohongshu (Little Red Book) (China) (CN)
ORCID
Venue
Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval
Publisher
ACM
Published
2024-07-10
Pages
2145-2155
DOI
10.1145/3626772.3657787
OpenAlex
W4400524854
Cited by
7
References
3
Citations by year
2026: 1 · 2025: 4 · 2024: 2
FWCI
1.45
Type
conference paper
Language
en
Open access
Licence
Locations
  • International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Quan Cheng Laboratory (QCLZD202301)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
Keywords
text-to-image generation, user capabilities, interaction logs, artistic creation, prompt quality

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review