Visual Conceptual Blending with Large-scale Language and Vision Models
Open accessAbstract
We ask the question: to what extent can recent large-scale language and image generation models blend visual concepts? Given an arbitrary object, we identify a relevant object and generate a single-sentence description of the blend of the two using a language model. We then generate a visual depiction of the blend using a text-based image generation model. Quantitative and qualitative evaluations demonstrate the superiority of language models over classical methods for conceptual blending, and of recent large-scale image generation models over prior models for the visual depiction.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Less work has been done to rigorously examine how users can prompt generative frameworks with natural language for visual generation purposes.
“While momentum has started to build in prompt engineering for text generation purposes, less work has been done to rigorously examine how users can prompt generative frameworks with natural language for visual generation purposes, which is the focus in this paper.”
From Design Guidelines for Prompt Engineering Text-to-Image Generative Models (2022) · cites this paper as [17]
The Introduction (p. 1) and Related Work (p. 2) describe using prompt engineering and textual descriptions to generate images, so the paper addresses natural-language-guided visual generation. It does not establish that less research has examined this than text-generation prompting, or examine how users formulate prompts. (LLM, conf=0.78)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067 · see citing works
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
The way we think: conceptual blending and the mind's hidden complexities
(2002) · Choice Reviews Online · doi:10.5860/choice.40-1223 · Cited by 5,328 · see citing works
Large Scale GAN Training for High Fidelity Natural Image Synthesis
Brock, A., Donahue, J., & Simonyan, K. (2018) · arXiv · doi:10.48550/arxiv.1809.11096 · Cited by 3,717 · see citing works
Generative Adversarial Text to Image Synthesis
Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., & Lee, H. (2016) · arXiv · doi:10.48550/arxiv.1605.05396 · Cited by 2,494 · see citing works
Zero-shot text-to-image generation
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021) · arXiv · doi:10.48550/arxiv.2102.12092 · Cited by 2,417 · see citing works
Generative Adversarial Networks
Drori, I. (2022) · Cambridge University Press eBooks · doi:10.1017/9781108891530.013 · Cited by 2,401 · see citing works
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834 · see citing works
Mental spaces : aspects of meaning construction in natural language
Fauconnier, G. (1994) · LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) · Cited by 1,691 · see citing works
How Can We Know What Language Models Know?
Jiang, Z., Xu, F. F., Araki, J., & Neubig, G. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00324 · Cited by 1,002 · see citing works
ConceptNet 5.5: An Open Multilingual Graph of General KnowledgePreprint
Speer, R., Chin, J., & Havasi, C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.03975 · Cited by 832 · see citing works
DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-To-Image Synthesis
Zhu, M., Pan, P., Chen, W., & Yang, Y. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00595 · Cited by 605 · see citing works
Context dependent recurrent neural network language model
Mikolov, T., & Zweig, G. (2012) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2012.6424228 · Cited by 570 · see citing works
Analogical Inference for Multi-Relational EmbeddingsPreprint
Liu, H., Wu, Y., & Yang, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.02426 · Cited by 295 · see citing works
DF-GAN: Deep Fusion Generative Adversarial Networks for Text-to-Image SynthesisPreprint
Tao, M., Tang, H., Wu, S., Sebe, N., Wu, F. Q., & Jing, X. (2020) · arXiv (Cornell University) · Cited by 121 · see citing works
Rewriting a Deep Generative Model
Bau, D., Liu, S., Tongzhou, W., Zhu, J.-Y., & Torralba, A. B. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58452-8_21 · Cited by 86 · see citing works
A computational framework for conceptual blending
Eppe, M., Maclean, E., Confalonieri, R., Kutz, O., Schorlemmer, M., Plaza, E., & Kühnberger, K. (2017) · Artificial Intelligence · doi:10.1016/j.artint.2017.11.005 · Cited by 67 · see citing works
Generating similes effortlessly like a Pro: A Style Transfer Approach for Simile Generation
Chakrabarty, T., Muresan, S., & Peng, N. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.524 · Cited by 48 · see citing works
Deep Convolutional Networks as Models of Generalization and Blending Within Visual CreativityPreprint
McCaig, G., DiPaola, S., & Gabora, L. M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.02478 · Cited by 22 · see citing works
Experiments with free concept generation in Divago
Pereira, F. C., & Cardoso, A. (2006) · Knowledge-Based Systems · doi:10.1016/j.knosys.2006.04.008 · Cited by 21 · see citing works
Vismantic: Meaning-making with Images.
Xiao, P., & Linkola, S. (2015) · Työväentutkimus Vuosikirja · Cited by 15 · see citing works
A Pig, an Angel and a Cactus Walk Into a Blender: A Descriptive Approach to Visual BlendingPreprint
Cunha, J. M., Gonçalves, J., Martins, P., Machado, P., & Cardoso, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.09076 · Cited by 14 · see citing works
The Good, the Bad, and the AHA! Blends.
Martins, P., Urbančič, T., Pollak, S., Nada Lavrač, A., & Cardoso, A. (2015) · ICCC · Cited by 11 · see citing works
El imperativo de la participación en la gestión patrimonial
Sánchez‐Carretero, C. (2019) · DIGITAL.CSIC (Spanish National Research Council (CSIC)) · doi:10.13039/501100000780 · Cited by 10 · see citing works
A computational model for visual conceptual blends
Karimi, P., Maher, M. L., Grace, K. S., & Davis, N. M. (2018) · IBM Journal of Research and Development · doi:10.1147/jrd.2018.2881736 · Cited by 10 · see citing works
Visual Hallucination For Computational Creation.
Berov, L., & Kühnberger, K. (2016) · ICCC · Cited by 8 · see citing works
Sbai, O., Couprie, C., & Aubry, M. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00438 · Cited by 7 · see citing works
Let's Figure This Out: A Roadmap for Visual Conceptual Blending.
Cunha, J. M., Martins, P., & Machado, P. (2020) · ICCC · Cited by 2 · see citing works
Flagged first, then by citations
1 of the 32 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- [Chilton, Petridis, and Agrawala 2019] Chilton, L. B.; Petridis, S.; and Agrawala, M. 2019. Visiblends: A flexible workflow for visual blends. In CHI .
- [Sitzmann et al. 2020] Sitzmann, V.; Martel, J.; Bergman, A.; Lindell, D.; and Wetzstein, G. 2020. Implicit neural representations with periodic activation functions. NeurIPS .
- [Steinbr¨ uck 2013] Steinbr¨ uck, A. 2013. Conceptual blending for the visual domain. Ph. D. dissertation, Masters thesis .
Cited by
Sorted by citations
Design Guidelines for Prompt Engineering Text-to-Image Generative Models
Liu, V., & Chilton, L. B. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501825 · Cited by 639 · see citing works
“[12] utilized text-to-image models to generate visual blends of concepts.”
MethodFill in the Blank: Context-aware Automated Text Input Generation for Mobile GUI Testing
Liu, Z., Chen, C., Wang, J., Che, X., Huang, Y.-K., Hu, J., & Wang, Q. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00119 · Cited by 130 · see citing works
AI’s Regimes of Representation: A Community-centered Study of Text-to-Image Models in South Asia
Qadri, R., Shelby, R., Bennett, C. L., & Denton, R. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594016 · Cited by 85 · see citing works
Design Guidelines for Prompt Engineering Text-to-Image Generative Models
Liu, V., & Chilton, L. B. (2021) · arXiv · doi:10.48550/arxiv.2109.06977 · Cited by 635 · see citing works
AI’s Regimes of Representation: A Community-Centered Study of Text-to-Image Models in South Asia
Qadri, R., Shelby, R. M., Bennett, C. L., & Denton, E. (2024) · doi:10.21428/2c646de5.2b73d5c2 · Cited by 4 · see citing works
- Authors
- Ge, S., & Parikh, D.
- Affiliations
- University of Maryland, College Park (US)
- Georgia Institute of Technology (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2106.14127
- OpenAlex
- W3175732208
- Cited by
- 4
- References
- 32
- Citations by year
- 2024: 1 · 2023: 2 · 2022: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- conceptual blending, large language models, text-to-image generation, generative image models, visual representation, multimodal models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
