Opal: Multimodal Image Generation for News Illustration
Open accessThe DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Abstract
Advances in multimodal AI have presented people with powerful ways to create images from text. Recent work has shown that text-to-image generations are able to represent a broad range of subjects and artistic styles. However, finding the right visual language for text prompts is difficult. In this paper, we address this challenge with Opal, a system that produces text-to-image generations for news illustration. Given an article, Opal guides users through a structured search for visual concepts and provides a pipeline allowing users to generate illustrations based on an article’s tone, keywords, and related artistic styles. Our evaluation shows that Opal efficiently generates diverse sets of news illustrations, visual assets, and concept ideas. Users with Opal generated two times more usable results than users without. We discuss how structured exploration can help users better understand the capabilities of human AI co-creative systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“As units, cells can then be assembled together into generation inputs (e.g., sentences into an essay and requirements into a prompt) which allows users to quickly assemble input variations [6,59] or to mix-and-match the variants [101,111].”
From Cells, Generators, and Lenses: Design Framework for Object-Oriented Interaction with Large Language Models (2023) · cites this paper as [59]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“The rise of transformer models [6] and its success in few-shot learning has led to the rise of a new form of interaction with AI models known as prompt engineering.”
BackgroundLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 1×
“[31] Many of the earliest open-source text-to-image generative frameworks to gain traction used CLIP in a discriminator like fashion, using it in conjunction with a host of generative models from BigGan to VQGAN.”
MethodKaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778
Analyzing and Improving the Image Quality of StyleGAN
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., & Aila, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00813 · Cited by 6,332
Analyzing and Improving the Image Quality of StyleGAN
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., & Aila, T. (2019) · arXiv · doi:10.48550/arxiv.1912.04958 · Cited by 6,332
Semantic Image Synthesis With Spatially-Adaptive Normalization
Park, T., Liu, M.-Y., Wang, T.-C., & Zhu, J.-Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00244 · Cited by 2,974
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.11487 · Cited by 2,528 · cited 3×
“For text-to-image generation, we used the checkpoint and configuration of VQGAN+CLIP that was pretrained on Imagenet with a codebook size of 16384 [37].”
BackgroundTaming Transformers for High-Resolution Image Synthesis
Esser, P., Rombach, R., & Ommer, B. (2020) · arXiv · doi:10.48550/arxiv.2012.09841 · Cited by 2,514
Research works printed but not linked: 16
- 2020. CHI ’20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu , HI, USA) . Association for Computing Machinery , New York, NY, USA . 2020. CHI ’20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA). Association for Computing Machinery, New York, NY, USA.
- Gunjan Aggarwal and Devi Parikh. 2020. Neuro-Symbolic Generative Art: A Preliminary Study. arxiv:2007.02171 [cs.AI] Gunjan Aggarwal and Devi Parikh. 2020. Neuro-Symbolic Generative Art: A Preliminary Study. arxiv:2007.02171 [cs.AI] · arXiv
- Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell Sandhini Agarwal Ariel Herbert-Voss Gretchen Krueger Tom Henighan Rewon Child Aditya Ramesh Daniel M. Ziegler Jeffrey Wu Clemens Winter Christopher Hesse Mark Chen Eric Sigler Mateusz Litwin Scott Gray Benjamin Chess Jack Clark Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever and Dario Amodei. 2020. Language Models are Few-Shot Learners. https://doi.org/10.48550/ARXIV.2005.14165 · DOI · Paper page
- Alex Calderwood , Vivian Qiu , Katy Ilonka Gero , and Lydia B Chilton . 2018 . How Novelists Use Generative Language Models: An Exploratory User Study. In 23rd International Conference on Intelligent User Interfaces. ACM. Alex Calderwood, Vivian Qiu, Katy Ilonka Gero, and Lydia B Chilton. 2018. How Novelists Use Generative Language Models: An Exploratory User Study. In 23rd International Conference on Intelligent User Interfaces. ACM.
- Angel X. Chang Mihail Eric Manolis Savva and Christopher D. Manning. [n.d.]. SceneSeer: 3D Scene Design with Natural Language. https://doi.org/10.48550/ARXIV.1703.00050 10.48550/ARXIV.1703.00050 Angel X. Chang Mihail Eric Manolis Savva and Christopher D. Manning. [n.d.]. SceneSeer: 3D Scene Design with Natural Language. https://doi.org/10.48550/ARXIV.1703.00050 · DOI · Paper page
- Bob Coyne and Richard Sproat. 2022. WordsEye: an automatic text-to-scene conversion system. https://doi.org/10.1145/383259.383316 · DOI · Paper page
- Leon A. Gatys Alexander S. Ecker and Matthias Bethge. 2015. A Neural Algorithm of Artistic Style. arxiv:1508.06576 [cs.CV] Leon A. Gatys Alexander S. Ecker and Matthias Bethge. 2015. A Neural Algorithm of Artistic Style. arxiv:1508.06576 [cs.CV] · arXiv
- Songwei Ge and Devi Parikh. 2021. Visual Conceptual Blending with Large-scale Language and Vision Models. arxiv:2106.14127 [cs.CL] Songwei Ge and Devi Parikh. 2021. Visual Conceptual Blending with Large-scale Language and Vision Models. arxiv:2106.14127 [cs.CL] · arXiv
- Hideaki Kawabata and Semir Zeki . 2004. Neural correlates of beauty.Journal of neurophysiology 91 4 ( 2004 ), 1699–705. Hideaki Kawabata and Semir Zeki. 2004. Neural correlates of beauty.Journal of neurophysiology 91 4 (2004), 1699–705.
- Vivian Liu and Lydia Chilton. [n.d.]. Neurosymbolic generation of 3D animal shapes through... - ceur-ws.org. http://ceur-ws.org/Vol-2903/IUI21WS-HAIGEN-8.pdf Vivian Liu and Lydia Chilton. [n.d.]. Neurosymbolic generation of 3D animal shapes through... - ceur-ws.org. http://ceur-ws.org/Vol-2903/IUI21WS-HAIGEN-8.pdf
- Tingting Qiao , Jing Zhang , Duanqing Xu , and Dacheng Tao . 2019. Learn , Imagine and Create: Text-to-Image Generation from Prior Knowledge . In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.). Vol. 32. Curran Associates , Inc .https://proceedings.neurips.cc/paper/ 2019 /file/d18f655c3fce66ca401d5f38b48c89af-Paper.pdf Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. 2019. Learn, Imagine and Create: Text-to-Image Generation from Prior Knowledge. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.). Vol. 32. Curran Associates, Inc.https://proceedings.neurips.cc/paper/2019/file/d18f655c3fce66ca401d5f38b48c89af-Paper.pdf
- Tingting Qiao Jing Zhang Duanqing Xu and Dacheng Tao. 2019. MirrorGAN: Learning Text-to-image Generation by Redescription. https://doi.org/10.48550/ARXIV.1903.05854 · DOI · Paper page
- Aditya Ramesh Prafulla Dhariwal Alex Nichol Casey Chu and Mark Chen. 2022. Hierarchical Text-Conditional Image Generation with CLIP Latents. https://doi.org/10.48550/ARXIV.2204.06125 · DOI · Paper page
- Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. https://doi.org/10.48550/ARXIV.1908.10084 · DOI · Paper page
- Shikhar Sharma Dendi Suhubdy Vincent Michalski Samira Ebrahimi Kahou and Yoshua Bengio. 2018. ChatPainter: Improving Text to Image Generation using Dialogue. https://doi.org/10.48550/ARXIV.1802.08216 · DOI
- Jiahui Yu Yuanzhong Xu Jing Yu Koh Thang Luong Gunjan Baid Zirui Wang Vijay Vasudevan Alexander Ku Yinfei Yang Burcu Karagol Ayan Ben Hutchinson Wei Han Zarana Parekh Xin Li Han Zhang Jason Baldridge and Yonghui Wu. 2022. Scaling Autoregressive Models for Content-Rich Text-to-Image Generation. https://doi.org/10.48550/ARXIV.2206.10789 · DOI · Paper page
Cited by
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · arXiv · doi:10.48550/arxiv.2304.03442 · Cited by 2,173
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187
“We synthesize this research to suggest that large language models, though not sufficient by themselves, open up a new angle for creating believable agents when leveraged using the appropriate architecture.”
BackgroundYenduri, G., Ramalingam, M., Selvi, G. C., Supriya, Y., Srivastava, G., Maddikunta, P. K. R., Raj, G. D., Jhaveri, R. H., Prabadevi, B., Wang, W., et al. (2024) · IEEE Access · doi:10.1109/access.2024.3389497 · Cited by 596
Brade, S., Wang, B., Sousa, M., Oore, S., & Grossman, T. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606725 · Cited by 198
Suh, S., Chen, M., Min, B., Li, T. J.-J., & Xia, H. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642400 · Cited by 148
Metadata
- Authors
- Liu, V., Qiao, H., & Chilton, L.
- Affiliations
- Columbia University (US)
- University of Toronto (CA)
- Venue
- Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology
- Publisher
- ACM
- Published
- 2022-10-28
- Pages
- 1-17
- OpenAlex
- W4307475428
- Cited by
- 104
- References
- 41
- Citations by year
- 2026: 11 · 2025: 32 · 2024: 36 · 2023: 25
- FWCI
- 26.33
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Symposium on User Interface Software and Technology (UIST) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation Graduate Research Fellowship Program (NSF DGE - 1644869)
- Topics
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Digital Media and Visual Art
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- text-to-image generation, news illustration, multimodal AI, human-AI co-creation, artistic style, user guidance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Opal: Multimodal Image Generation for News Illustration. https://app.cochat.ai/papers/10.1145/3526113.3545621
- The paper
- Liu, V., Qiao, H., & Chilton, L. (2022). Opal: Multimodal Image Generation for News Illustration. Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. https://doi.org/10.1145/3526113.3545621
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.