CoChat
Sign Up
PreprintCited by 38Multimodal Machine Learning ApplicationsGenerative Adversarial Networks and Image SynthesisCategorization, perception, and language

The DOI matches its DataCite record.

Testing Relational Understanding in Text-Guided Image Generation

Open access
Conwell, Colin, Ullman, Tomer·2022·arXiv
DOI 10.48550/arxiv.2208.00005arXiv 2208.00005
Open this paper in CoChat — free accountOpen PDF

Abstract

Relations are basic building blocks of human cognition. Classic and recent work suggests that many relations are early developing, and quickly perceived. Machine models that aspire to human-level perception and reasoning should reflect the ability to recognize and reason generatively about relations. We report a systematic empirical examination of a recent text-guided image generation model (DALL-E 2), using a set of 15 basic physical and social relations studied or proposed in the literature, and judgements from human participants (N = 169). Overall, we find that only ~22% of images matched basic relation prompts. Based on a quantitative examination of people's judgments, we suggest that current image generation models do not yet have a grasp of even basic relations involving simple objects and agents. We examine reasons for model successes and failures, and suggest possible improvements based on computations observed in biological intelligence.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2208.00005

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • DALL-E 2 performed poorly when prompted to generate a scene with objects arranged in unfamiliar ways.

    “Conwell and Ullman (2022) found that the recent image generation model DALL-E 2 performed poorly when prompted to generate a scene with objects arranged in unfamiliar ways.”

    From Consciousness in Artificial Intelligence: Insights from the Science of Consciousness (2023) · cites this paper as Conwell and Ullman 2022

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Still being checked.

Research works printed but not linked: 38

  1. [1][1] Hespos SJ, Spelke ES. Conceptual precursors to language. Nature. 2004;430(6998):453-6. -
  2. [2][2] Talmy L. Lexicalization patterns: Semantic structure in lexical forms. Language typology and syntactic description. 1985;3(99):36-149. -
  3. [3][3] Hafri A, Firestone C. The perception of relations. Trends in Cognitive Sciences. 2021;25(6):47592. -
  4. [4][4] Saharia C, Chan W, Saxena S, Li L, Whang J, Denton E, et al. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. arXiv preprint arXiv:220511487. 2022. -
  5. [5][5] Ramesh A, Dhariwal P, Nichol A, Chu C, Chen M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:220406125. 2022. -
  6. [6][6] Marcus G, Davis E, Aaronson S. A very preliminary analysis of DALL-E 2. arXiv preprint arXiv:220413807. 2022. -
  7. [8][8] Farid H. Perspective (In) consistency of Paint by Text. arXiv preprint arXiv:220614617. 2022. -
  8. [9][9] Liu N, Li S, Du Y, Torralba A, Tenenbaum JB. Compositional Visual Generation with Composable Diffusion Models. arXiv preprint arXiv:220601714. 2022. -
  9. [10][10] Nichol A, Dhariwal P, Ramesh A, Shyam P, Mishkin P, McGrew B, et al. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:211210741. 2021. -
  10. [11][11] Crowson K, Biderman S, Kornis D, Stander D, Hallahan E, Castricato L, et al. Vqgan-clip: Open domain image generation and editing with natural language guidance. arXiv preprint arXiv:220408583. 2022. -
  11. [12][12] Rombach R, Blattmann A, Lorenz D, Esser P, Ommer B. High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2022. p. 10684-95. -
  12. [13][13] Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, et al. Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning. PMLR; 2021. p. 8748-63. -
  13. [14][14] Chen L. Topological structure in visual perception. Science. 1982;218(4573):699-700. -
  14. [15][15] Lovett A, Franconeri SL. Topological relations between objects are categorically coded. Psychological science. 2017;28(10):1408-18. -
  15. [16][16] Strickland B, Keil F. Event completion: Event based inferences distort memory in a matter of seconds. Cognition. 2011;121(3):409-15. -
  16. [17][17] Kellman PJ, Spelke ES. Perception of partly occluded objects in infancy. Cognitive psychology. 1983;15(4):483-524. -
  17. [18][18] Spelke E. Initial knowledge: Six suggestions. Cognition. 1994;50(1-3):431-45. -
  18. [19][19] Yildirim I, Siegel MH, Tenenbaum JB. Perceiving fully occluded objects via physical simulation. In: Proceedings of the 38th annual conference of the cognitive science society; 2016. . -
  19. [20][20] Gao T, Newman GE, Scholl BJ. The psychophysics of chasing: A case study in the perception of animacy. Cognitive psychology. 2009;59(2):154-79. -
  20. [21][21] Hamlin JK, Wynn K, Bloom P. Social evaluation by preverbal infants. Nature. 2007;450(7169):557-9. -
  21. [22][22] Ullman T, Baker C, Macindoe O, Evans O, Goodman N, Tenenbaum J. Help or hinder: Bayesian models of social goal inference. Advances in neural information processing systems. 2009;22. -
  22. [23][23] van Buren B, Uddenberg S, Scholl BJ. The automaticity of perceiving animacy: Goal-directed motion in simple shapes influences visuomotor behavior even when task-irrelevant. Psychonomic bulletin & review. 2016;23(3):797-802. -
  23. [24][24] Glanemann R, Zwitserlood P, Bölte J, Dobel C. Rapid apprehension of the coherence of action scenes. Psychonomic bulletin & review. 2016;23(5):1566-75. -
  24. [25][25] Dobel C, Gumnior H, Bölte J, Zwitserlood P. Describing scenes hardly seen. Acta psychologica. 2007;125(2):129-43. -
  25. [26][26] Guan C, Firestone C. Seeing what's possible: Disconnected visual parts are confused for their potential wholes. Journal of experimental psychology: general. 2020;149(3):590. -
  26. [27][27] Firestone C, Scholl B. Seeing physics in the blink of an eye. Journal of Vision. 2017;17(10):2033. -
  27. [28][28] Hafri A, Bonner MF, Landau B, Firestone C. A phone in a basket looks like a knife in a cup: The perception of abstract relations. PsyArXiv. 2020. -
  28. [29][29] Ehrhardt S, Groth O, Monszpart A, Engelcke M, Posner I, Mitra N, et al. RELATE: Physically plausible multi-object scene synthesis using structured latent spaces. Advances in Neural Information Processing Systems. 2020;33:11202-13. -
  29. [30][30] Johnson J, Hariharan B, Van Der Maaten L, Fei-Fei L, Lawrence Zitnick C, Girshick R. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In: Proceedings of the IEEE conference on computer vision and pattern recognition; 2017. p. 2901-10. -
  30. [31][31] Peer E, Brandimarte L, Samat S, Acquisti A. Beyond the Turk: Alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology. 2017;70:15363. -
  31. [32][32] Pelletier FJ. The principle of semantic compositionality. Topoi. 1994;13(1):11-24. -
  32. [33][33] Pelletier FJ. Semantic compositionality. In: Oxford research encyclopedia of linguistics; 2016. . -
  33. [34][34] Tai KS, Socher R, Manning CD. Improved semantic representations from tree-structured long short-term memory networks. arXiv preprint arXiv:150300075. 2015. -
  34. [35][35] Summerfield C, Luyckx F, Sheahan H. Structure learning and the posterior parietal cortex. Progress in neurobiology. 2020;184:101717. -
  35. [36][36] Shridhar M, Manuelli L, Fox D. CLIPort: What and Where Pathways for Robotic Manipulation. In: Proceedings of the 5th Conference on Robot Learning (CoRL); 2021. . -
  36. [37][37] Steinberg J, Sompolinsky H. Associative memory of structured knowledge. bioRxiv. 2022. -
  37. [38][38] Silver RA. Neuronal arithmetic. Nature Reviews Neuroscience. 2010;11(7):474-89. -
  38. [39][39] Fusi S, Miller EK, Rigotti M. Why neurons mix: high dimensionality for higher cognition. Current opinion in neurobiology. 2016;37:66-74.

Cited by

Sorted by citations

Related

Metadata

Authors
Conwell, C., & Ullman, T.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2208.00005
arXiv
2208.00005
OpenAlex
W4289645295
Cited by
36
References
0
Citations by year
2026: 1 · 2025: 2 · 2024: 22 · 2023: 8 · 2022: 3
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • OSF Preprints (OSF Preprints) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Categorization, perception, and language
Keywords
relational understanding, text-to-image generation, DALL-E 2, physical relations, interpersonal relationships, relation recognition, generative AI, human-AI comparison, biological intelligence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review